The Pascal Triangle of a Discrete Image: Definition, Properties and Application to Shape Analysis

We define the Pascal triangle of a discrete (gray scale) image as a pyramidal arrangement of complex-valued moments and we explore its geometric significance. In particular, we show that the entries of row k of this triangle correspond to the Fourier series coefficients of the moment of order k of the Radon transform of the image. Group actions on the plane can be naturally prolonged onto the entries of the Pascal triangle. We study the prolongation of some common group actions, such as rotations and reflections, and we propose simple tests for detecting equivalences and self-equivalences under these group actions. The motivating application of this work is the problem of characterizing the geometry of objects on images, for example by detecting approximate symmetries.

We can express the relationship between the moments and the image I in matrix form: where and Z † is the conjugate transpose of Z. Observe that Z is a Vandermonde matrix and therefore is invertible when the pixel locations z k are pairwise distinct. Therefore, if the pixel coordinates are known and pairwise distinct, one can reconstruct the image I by matrix inversion: W = (Z −1 ) † τ N (I)Z −1 .
Proof . Recall the definition of the moments µ j,l = N k=1 z j kz l k ρ(z k ).
We consider the vector formed by the moments {µ j,0 } N −1 j=0 , which can be written in matrix form as  . . .
Observe that the coefficient matrix in (2) is a Vandermonde matrix. The Vandermonde matrix has full rank when z j = z k for all distinct j, k = 1, 2, . . . , N . Thus, since the pixel locations are assumed to be distinct, we can reconstruct the pixel intensities {ρ(z k )} N k=1 by inverting the coefficient matrix and multiplying by the moment vector on the left-hand-side.
Notice that if we consider the Pascal triangle T N (I) of order N , then knowledge of the second right diagonal row of T N (I), i.e. {µ j,1 } N −1 j=0 , is also sufficient for image reconstruction as long as the z k 's are pairwise distinct and nonzero. This is because the vector formed by the moments {µ j,1 } N −1 j=0 can be written in matrix form as        µ 0,1 µ 1,1 µ 2,1 . . .
The coefficient matrix in (3) is a Vandermonde matrix multiplied by a diagonal matrix. Assuming that the pixel locations are pairwise distinct insures that the Vandermonde matrix is invertible, and further assuming that they are nonzero insures invertibility of the diagonal matrix. Hence the coefficient matrix in (3) is nonsingular and we can reconstruct the pixel intensities {ρ(z k )} N k=1 from T N (I) by inverting this coefficient matrix and multiplying by the moment vector on the left-hand-side.
A similar argument can be used to show that, for any fixed l, the pixel intensities can be reconstructed from the moment vector {µ j,l } N −1 j=0 , which can be obtained from the Pascal triangle T N +l−1 (I) of order N + l − 1.
Remark 1. In practice, when reconstructing the pixel intensities of an image I, floating point errors in the matrix inversion can result in inaccuracies in the reconstructed image. In fact, the recovered pixel intensities may be complex valued. While the imaginary part of the result tends to be quite small, it is advantageous to first reformulate the problem to guarantee a real solution. One way to force the solution to be real is to separate equation (2) into two sets of equations with real coefficients. More specifically, we can separate the equation system into its real part and its imaginary part, and combine these two real equation systems into one. After this, a real solution for the new equation system can be found, for example, by singular value decomposition (SVD). Lemma 2. Given the moments matrix τ N (I) of a discrete image I and an upper bound on the number N of pixels, one can reconstruct the pixel location z k and the intensity ρ(z k ) for all z k such that ρ(z k ) = 0. 3 Proof . If the number of pixels in the image I is strictly less than N , we can extend I to an image with N pixels by adding zero intensity pixels. Without loss of generality, we assume that ρ(z k ) = 0 for k = 1, . . . , s and ρ(z k ) = 0 for k = s + 1, . . . , N . Consider the polynomial where the coefficients c j are polynomials in the z k 's.
Thus τ s (I) is invertible, since the locations z k are pairwise distinct and the pixel intensities ρ(z k ) are nonzero. Hence we can solve the above equation system for (c s , c s−1 , . . . , c 1 ) by inverting τ s (I) and multiplying by the vector on the right-hand-side of equation (4).
Since the c k 's determine the polynomial P (t), we can solve for the roots of P (t) = 0, which are actually {z k } s k=1 . By Lemma 1, we can subsequently obtain the pixel intensities {ρ(z k )} s k=1 .

Remark 2.
To determine the number of nonzero pixels, we can look at the rank of τ N (I). Since τ N (I) = Z † W Z by equation (1) and rank(Z † ) = rank(Z) = N , rank(W ) = s, we can conclude that rank(τ N (I)) = s.
Since the Pascal triangle T 2N −2 (I) of the image I contains all the information needed to recover τ N (I), we have the following corollary: Corollary 1 (image reconstruction property). Given the Pascal triangle T 2N −2 (I) of a discrete image I, one can reconstruct both the grid point locations {z k } N k=1 and the corresponding intensities {ρ(z k )} N k=1 for all those z k such that ρ(z k ) = 0.

Relationship with the Radon transform
The Radon transform f θ (r) is the projection of the image I = {(z k , ρ(z k ))} N k=1 onto the straight line through the origin with direction vector cos(θ) sin(θ) T , i.e.
where S = {k | x k cos(θ) + y k sin(θ) = r, k = 1, 2, . . . , N }. Since f θ (r) is a periodic function of θ with period 2π, any of its n-th order moment m n (θ) is also periodic with period 2π. It turns out that, for any n = 0, 1, 2, . . ., the coefficients of the Fourier series of m n (θ) are given by the entries of row (n + 1) of T r (I) with r ≥ n.
There is a unique solution for µ 0,n , . . . , ( n l ) µ l,n−l , . . . , µ n,0 if and only if since e −inθ j = 0, ∀ j = 1, . . . , n + 1. Observe that the above determinant is a Vandermonde determinant. It is nonzero if e i2θ j = e i2θ k for all distinct j, k = 1, . . . , n + 1. Therefore, if the θ j 's are such that e i2θ j = e i2θ k (thus the need to pick a generic sample set), we will get a unique solution for µ 0,n , . . . , ( n l ) µ l,n−l , . . . , µ n,0 . Hence we can reconstruct the last row of T n (I).

Corollary 2. Given the grid point locations {z
Proof . From the given radon transform of the image at different angles, we can calculate the moments m n (θ k ) | n = 0, . . . , N − 1, k = 1, . . . , n + 1 . The conclusion followed by combining Lemmas 1 and 4.
The diagram of Fig. 3 thus commutes. Note that, one could use a similar argument along with Corollary 1 to show that Radon transform

Prolongation of group actions on the moments and invariantization
Let (G, ·) be a group acting on the complex plane: This induces a group transformation (G, •) of the discrete image I = {(z k , ρ(z k ))} N k=1 , namely Then the induced transformation (G, * ) on moments {µ j,l } j,l∈Z ≥ 0 is In other words, the transformed moments are the moments of the transformed image.
Example 1. Consider the action of G = C on C by translation Then the induced transformation on the image In other words, the image is translated horizontally with distance x 0 = Re(z 0 ) and vertically with y 0 = Im(z 0 ). The transformed complex moments arẽ Written in matrix form, the transformation of the moment matrix Having obtained an explicit formula for the action of G on the moments, we follow Fels and Olver's moving frame method [1, 2, 7] to obtain a set of invariant functions of the moments. More specifically, we consider the cross-section defined byμ 1,0 = 0. The group transformation that maps τ N (I) to the cross-section is the moving frame z 0 = − µ 1,0 µ 0,0 . By applying the moving frame to the moment matrix, we obtain the matrixτ . By equivariance of the moving frame, all the entries ofτ N (I) are invariant under translation. One can check that theses entriesμ j,l are actually the centralized moments By normalizing (i.e. applying the moving frame transformation to) the coordinates of T r (I), we obtain the translation invariant Pascal triangle T r trans (I) for a discrete image I: Observe that the corresponding n-th order central momentm n (θ) of the image, where r 0 (θ) = x 0 cos θ + y 0 sin θ is the projection of the centroid, is invariant under translations.
Lemma 5 (orbit separation property of T 2N −2 trans (I)). Let I 1 , I 2 be two discrete gray scale images with the same number N of pixels. There exists a translation g ∈ C such that g , for some z 0 ∈ C, k = 1, . . . , N . From equation (7) we know that Then applying equation (8), we can get for any j, l ∈ Z ≥0 Therefore T r trans (I 1 ) = T r trans (I 2 ) for any r ∈ Z ≥0 . ⇐ If T r trans (I 1 ) = T r trans (I 2 ) for r ≥ 2N − 2, from Corollary 1, we conclude that I trans for any k = 1, 2, . . . , N . Without loss of generality, we assume that k,trans and ρ 1 z Remark 3. Without loss of generality, we can assume that the two images have the same number of pixels by simply adding zero valued pixels to the smaller image.
Example 2. Consider the action of G = R + on C by scaling Then the induced transformation on the image In other words, the image is scaled by a factor λ both horizontally and vertically. Then the transformed complex moments arê Written in matrix form, the moment matrix for the new imageÎ after scaling is Again, we use the moving frame method of Fels and Olver to obtain a set of invariant functions of the moments. Notice x 2 k +ŷ 2 k ρ(ẑ k ) = 0 unless all ρ(z k ) are zero. We consider the cross-section defined byμ 1,1 = 1. The group transformation that maps τ N (I) to the cross-section is the moving frame λ = (µ 1,1 ) − 1 2 . By applying the moving frame to the moment matrix, we obtain the matrix By equivariance of the moving frame, all the entries ofτ N (I) are invariant under scaling. By normalizing the coordinates of T r (I), we obtain the scaling invariant Pascal triangle T r scale (I) for a discrete image I: Observe that the corresponding n-th order normalized momentm n (θ) = m n (θ)/µ n 2 1,1 is invariant under scaling.
Lemma 6 (orbit separation property of T 2N −2 scale (I)). Let I 1 , I 2 be two discrete gray scale images with the same number N of pixels. There exists a scaling g ∈ R + such that g Proof . ⇒ If ∃ g ∈ G such that g • I 1 = I 2 , we have z for some λ ∈ R + , k = 1, . . . , N . Since for I 1 and I 2 , the corresponding scaling invariant moments arê k,scale = λ 2 z More generally, consider the action of group G of diagonal matrices on R 2 + by scaling Then the induced transformation on the image I = {(z k , ρ(z k ))} N k=1 is In other words, the image is scaled by a factor λ 1 horizontally and scaled by λ 2 vertically.
Notice that after the transformation, the pixel coordinates becomê Then we have the transformed complex momentŝ which is a linear combination of the last row of the Pascal triangle T j+l (I).
Example 3. Consider the action of G = {z ∈ C||z| = 1} on C by rotation Then the induced transformation on the image I = {(z k , ρ(z k ))} N k=1 is In other words, the image is rotated counterclockwise with an angle θ 0 . The transformed complex moments are Written in matrix form, the moment matrix for the new image I after rotation is The Pascal Triangle of a Discrete Image 13 Now we will use the moving frame method of Fels and Olver to obtain a set of invariant functions of the moments. If µ 0,2 = 0, we normalize the imaginary part of µ 0,2 to zero by specifying the rotation angle θ 0 . Since µ 0,2 = µ 0,2 e −i2θ 0 , looking at µ 0,2 as a vector in R 2 representing the complex number µ 0,2 , we set 2θ 0 = ( µ 0,2 , e 1 ) + 2kπ, k ∈ Z.
Lemma 7 (orbit separation property of T 2N −2 rotate (I) ). Let I 1 , I 2 be two discrete gray scale images with the same number N of pixels. There exists a rotation g ∈ {z ∈ C||z| = 1} such that g • I 1 = I 2 , where • is defined as in (10) ⇐⇒ T r rotate (I 1 ) = T r rotate (I 2 ) for r ≥ 2N − 2.
In conclusion, we have the translation, scaling and rotation invariant Pascal triangle T r int (I):  5 Geometric interpretation of the moments
Equation (14) indicates that the quantity |μ 0,2 | µ 1,1 prescribes the relationship between the maximum and the minimum values of the standard deviation of the random transform. It thus gives us a quantification of the elongation of the shape illustrated by the image 5 .
The case |μ 0,2 | µ 1,1 = 1 corresponds to the most extreme elongation, namely the straight lines. Proof . ⇒ Suppose we have a straight line. As the line is put vertically, the projection of the line is a dot. Hence the second moment of the Radon transform is zero at that angle θ * , i.e.
Let the Radon transform fθ(r) at angleθ be a discrete function. Without loss of generality, we assume that N k=1 fθ(r k (θ)) = 1 and fθ(r k (θ)) ≥ 0. Sincẽ where r 0 (θ) is the projection of the centroid of the image, we observe that fθ(r) = δ(r − r 0 (θ)). Then we conclude that the image lies on a line through r 0 (θ) with angleθ + π 2 to the x-axis.
The other extreme case is when |μ 0,2 | µ 1,1 = 0. This corresponds tom 2 (θ) = const. So the standard deviation of the projection is the same for all directions. There are many ways for this to happen. One interesting case is the discrete analogue of rotation symmetries.

Definition 2. An object is said to haveÑ -fold rotation symmetry (Ñ -FRS) if it is unchanged by a rotation around its centroid by 2kπ
N , for all k = 1, . . . ,Ñ .
One can give a statistical interpretation of our proposed shape elongation measure |μ 0,2 | µ 1,1 . Indeed after renormalizing the total ink µ 0,0 = N k=1 ρ(z k ) to one, one can view the pixel intensities as describing a discrete probability distribution. The standard deviation matrix of that distribution is then determined by the third row of the Pascal triangle, as stated in the following lemma: Lemma 11. Consider the discrete image I as a bivariate distribution with the joint probability mass function P (X = x k − x 0 , Y = y k − y 0 ) = ρ(z k ) µ 0,0 , k = 1, 2, . . . , N . The covariance matrix Σ of that distribution is given by Proof . Observe that Since the random variables X, Y both have zero mean, we have Then one can check that Recall that, in order to obtain the standard deviation m 2 (θ) of the projection of a bivariate distribution onto the line with direction vector (x, y) = r(cos θ, sin θ), one can simply project the standard deviation matrix Σ onto (x, y) : cos θ sin θ Σ cos θ sin θ = m 2 (θ).
It is easy to check that the relationship between the shape elongation |μ 0,2 | µ 1,1 and the eigenvalues λ max , λ min of the standard deviation matrix Σ is

Rotational symmetry
We have seen in the last section that an image I having anÑ -FRS hasμ 0,2 = 0. More generally, we have the following lemmas, which was used in [5] as the basis for a HAZMAT sign recognition method.
Lemma 12. LetÑ be a (finite) integer. If an image I has anÑ -fold rotation symmetry and if l−j N is not an integer, thenμ j,l = 0. Conversely, ifμ j,l = 0 for all l−j N that are not an integer, then the image I has anÑ -fold rotation symmetry 6 .
Proof . ⇒ Let us rotate I clockwise around the origin by 2π N . Due to its symmetry, the rotated image I must be the same as the original one. In particular, it must hold µ j,l = e 2πi(l−j)/Ñμ j,l =μ j,l .
Since l−j N is not an integer, this equation can be fulfilled only ifμ j,l = 0. ⇐ Supposeμ j,0 = 0 for any j N not an integer of some finite integerÑ . Let us rotate I clockwise around the origin by 2kπ N for each k = 1, 2, . . . ,Ñ − 1, thenμ j,0 = e −2πijk/Ñμ j,0 for each k and all j = 0, 1, . . . , N − 1. For jk N ∈ Z, it is easy to check that e −2πijk/Ñ = 1, hencẽ µ j,0 =μ j,0 . For jk N / ∈ Z, j N is not an integer either. Thenμ j,0 = 0 =μ j,0 . In this way we havẽ µ j,0 =μ j,0 for all j = 0, 1, . . . , N − 1. Since in the proof of Lemma 1, we showed that {μ j,0 } N −1 j=0 uniquely determine I, then we can conclude that I and I are the same for any rotation with angle 2kπ N , k = 1, 2, . . . ,Ñ − 1. Therefore the image I has anÑ -fold rotation symmetry. Remark 4. As N increases, more and more columns of the Pascal triangle T r (I) become zero, so that in the limit case, as N → ∞, all entries µ j,l with j = l of T r (I), for any order r, vanish. This limit case corresponds to ∞-fold rotation symmetry (e.g. circles), which however does not occur among discrete images.

Ref lection symmetry
Consider the reflection of an image about the line through the origin with direction vector cos(θ) sin(θ) T , θ ∈ (− π 2 , π 2 ]. The point z k = x k +iy k is mapped to z k =z k e i2θ = (x k cos(2θ)+ y k sin(2θ)) + i(x k sin(2θ) − y k cos(2θ)) under the reflection with its pixel intensity ρ(z k ) staying the same. Then the new complex moment is Therefore the moment matrix for the new image I after reflection is We can conclude from the above relation that if an image is symmetric with respect to the x-axis (i.e. θ = 0), then we will have τ N (I) = τ N (I) T , i.e. µ j,l = µ l,j for all j, l ∈ Z ≥0 . Since µ j,l =μ l,j by definition, this means that all the µ j,l 's are real 7 .
Similarly, if an image is symmetric with respect to the y-axis (i.e. θ = π 2 ), we can conclude that µ j,l 's are real for j, l of the same parity and µ j,l 's are imaginary for j, l of opposite parity.

Experiments and results
To illustrate the application of the Pascal triangle to symmetry detection, we used our results to design a simple reflection symmetry detection method 8 . We tested this method on images from the MPEG-7 CE Shape-1 Part-B data set 9 . The data set includes 1400 binary images. The images are divided into 70 object classes, each object class containing 20 images. All our ground truth data and classification results can be downloaded from https://engineering. purdue.edu/~mboutin/symmetric_shapes.

Horizontally symmetric object detection experiment
In this experiment, we identified images that have a horizontal axis of reflection symmetry using only the first four rows of the Pascal triangle. Our data set consists of 320 shapes from the MPEG-7 shape database. Specifically, we included all 20 images contained in each of the following 16 classes: Bird, Device1-Device5, Device7-Device9, Watch, Cup, Dog, Flatfish, Glas, Hat, Tree. We first manually divided the data into two sets. One set, called Set 1a, was assigned all objects that appeared to have a clear horizontal symmetry axis, up to some minor details. The remaining set, called Set 2a, was assigned the remaining images. Set 1a and Set 2a contain 113 and 207 images respectively. As one can see by inspecting Set 2a (see for example the images in Fig. 5(c) and (d)), our classification was quite strict. Indeed, we excluded many objects that could be declared symmetric under a greater tolerance for error. We thus created a second grouping allowing for more errors: Set 1b and Set 2b, which are the data sets resulting from this more lenient definition of symmetry.
Recall that, by Lemma 13, an image has a horizontal axis of symmetry if and only if all the entries of its Pascal triangle are real. Since we are focusing on the symmetry of the object contained in the image, as opposed to the image itself, we need to consider the translation invariant Pascal triangle consisting of the centralized momentsμ j,l . Thus horizontally symmetric objects should be recognizable by considering the magnitude of the imaginary part of each of its centralized moments. Note thatμ 0,0 andμ 0,1 are always real and thatμ 2,0 =μ 0,2 . Thus, if we restrict ourselves to the first four rows of the Pascal triangle, for simplicity, then horizontally symmetric objects are characterized by the fact thatμ 0,2 ,μ 0,3 andμ 1,2 are real. In other words, objects that are approximately symmetric should haveμ 0,2 ,μ 0,3 andμ 1,2 with an imaginary part close to zero. In order to remove the scale ambiguity resulting from the arbitrary scale used to describe the pixel coordinates, we followed the approach described in Section 4, Example 2 to invariantize our coordinates with respect to scaling. Our specific classification criteria were: where r is a variable threshold.
In our experiments, we varied the threshold r from 0.005 to 0.15. For each value of r, we classified every image as either "symmetric" or not symmetric using the above mentioned 8 The reader interested in the application of the Pascal triangle to the detection of rotational symmetries is invited to read [5].
9 Shape data for the MPEG-7 core experiment CE-Shape-1, http://www.cis.temple.edu/~latecki/ TestData/mpeg7shapeB.tar.gz.  The results obtained when using the data sets Set 1a and Set 2a are plotted in Fig. 4(a), and those obtained using Set 1b and Set 2b are plotted in Fig. 4(b). Observe that the maximum accuracy for the first data set, 83.75% (obtained around r = 0.07), goes up to 96.25% (obtained around r = 0.11) for the second data set. This is consistent with the fact that the second data set was constructed using a greater tolerance for error: after all, we are only using the first four rows of the triangle to classify the shape. Indeed, the shapes that were moved from Set 2a to Set 1b caused the number of false positive to decrease and thus the precision to increase correspondingly. Fig. 5 illustrates some of our results.

Symmetric objects (any axis) detection experiment
In this experiment we identified images that have an axis of reflection symmetry. Our data set consists of 200 shapes from the following 10 classes of the MPEG-7 shape database: Beetle, Bell, Bird, Butterfly, Camel, Cattle, Classic, Crown, Horseshoe, Lizzard. Re(μ 3,4 ) and T is a variable threshold.
In our experiments, we varied the threshold T from 1 • to 15 • . For each value of T , we classified every image as either "symmetric" or not symmetric using the above mentioned criteria. And for each symmetric image, we found its symmetry axis. We also computed the precision, recall, and accuracy for each value of T .
The classification results obtained when using the data sets Set 1 and Set 2 are plotted in Fig. 6. Observe that the maximum accuracy for the data set is 79.5% at T = 4 • . The accuracy of the experiment could be improved by using more moments: after all, we are only using three moments to classify the shapes. Indeed, using more moments would yield a more selective criterion, which should decrease the number of false positives and increase the number of true negatives. Fig. 7 illustrates some of our results.

Conclusion and future work
We have introduced the Pascal triangle of a discrete image, which is constructed using complexvalued moments. We obtained the relationship between the triangle and the Fourier series coefficients of the moment of the Radon transform of the image, that is, each row n of the Pascal triangle contains the coefficients of the Fourier series of the n-th order moment of the Radon transform of the image. This relationship gives the moments a clear geometric interpretation. For example, µ 0,3 8 of an image is the coefficient of e i3θ in the third order moment m 3 (θ) of the Radon transform of the image, and thus when |µ 0,3 | is large, then the order three variation of the skewness of the projection is proportionally large.
We showed that the image can be fully reconstructed using a finite number of rows of the triangle. This fact, which is specific to discrete (finite) images, allows us to be able to derive necessary and sufficient conditions for the presence of various symmetries. It also allows us to conclude that the invariantized Pascal triangle separates the orbits of certain group actions. Indeed, by using the moving frame method we were able to invariantize the Pascal triangle with respect to translation, rotation and scaling, and by using the reconstruction property of the invariantized Pascal triangle, we were able to show the uniqueness of the reconstruction modulo these transformations.  We tested the application of the Pascal triangle to the recognition of symmetric shapes from the MPEG-7 shape database. More specifically, we derived a simple method to detect horizontal symmetries using the first four rows of the triangle. We then tested this method using 16 object classes. We also derived a simple method to detect symmetry axes in objects using entries within only the first eight rows of the triangle. We then tested this method using 10 object classes.
Extension of our method to other group actions such as affine transforms should be doable using the moving frame method. Observe that our definition for µ j,l naturally extends to vector valued pixel intensities. Therefore, it should be straightforward to extend our framework to the case of color images. We are looking forward to also extending this work to the case of 3D objects.