Linear neighborhood operators are fundamental tools for a large part of image processing. We present a method that treats such operators within a unified framework. This framework enables one to treat linear combinations, concatenations, resolution changes, or rotations of operators in a canonical manner. Various families of operators with special kinds of symmetries (such as translation, rotation, magnification) are explicitly constructed in 1-D, 2-D, and 3-D. A concept of "order" is defined, and finite orthonormal bases of functions closely connected with the operators of various orders are constructed. Linear transformations between the various representations are considered. One particular representation enables one to compile differential geometrical expressions immediately into local algorithms (e.g., for edge curvature). In this representation, the "order" is an order of derivation in a certain precisely defined sense. The method is based on two fundamental assumptions: First, a decrease of resolution should not introduce spurious detail (the basic requirement for a "scale space"), and second, the local operators should be self-similar under changes of resolution. These assumptions merely sum up the even more general need for homogeneity, isotropy, scale invariance, and separability of independent dimensions of front end processing in the absence of a priori information.