
SIGMA 2 (2006), 039, 21 pages math.OC/0604220
http://dx.doi.org/10.3842/SIGMA.2006.039
Combined ReducedRank Transform
Anatoli Torokhti and Phil Howlett
School of Mathematics and Statistics, University of South Australia, Australia
Received November 25, 2005, in final form March 22, 2006; Published online April 07, 2006
Abstract
We propose and justify a new approach to constructing optimal
nonlinear transforms of random vectors.
We show that the proposed transform improves such characteristics of rankreduced transforms
as compression ratio, accuracy of decompression and reduces required computational work.
The proposed transform T_{p} is presented in the form of a sum with p terms where each
term is interpreted as a particular rankreduced transform. Moreover, terms in T_{p}
are represented as a combination of three operations F_{k},
Q_{k} and φ_{k} with
k = 1,...,p.
The prime idea is to determine F_{k} separately,
for each k = 1,...,p, from an associated
rankconstrained minimization problem similar to that used in the KarhunenLoève
transform. The operations Q_{k}
andφ_{k} are auxiliary for finding F_{k}. The
contribution of each term in T_{p} improves the entire transform performance.
A corresponding unconstrained nonlinear optimal transform is also considered. Such a
transform is important in its own right because it is treated as an optimal filter without
signal compression.
A rigorous analysis of errors associated with the proposed transforms is given.
Key words:
best approximation; Fourier series in Hilbert space; matrix computation.
pdf (428 kb)
ps (235 kb)
tex (24 kb)
References
 Hotelling H., Analysis of a complex of statistical variables
into Principal Components, J. Educ. Psychol., 1933, V.24, 417441, 498520.
 Karhunen K., Über Lineare Methoden in der
Wahrscheinlichkeitsrechnung, Ann. Acad. Sci. Fennicae, Ser. A, 1947, V.137.
 Loève M., Fonctions aléatoires de second order, in P. Lévy,
Processus Stochastiques et Mouvement Brownien, Paris, Hermann, 1948.
 Jolliffe I.T., Principal component analysis, New York, Springer Verlag,
1986 (2 ed., 2002).
 Scharf L.L., The SVD and reduced rank signal processing,
Signal Processing, 1991, V.25, 113133.
 Yamashita Y., Ogawa H., Relative KarhunenLoéve transform,
IEEE Trans. on Signal Processing, 1996, V.44, 371378.
 Hua Y., Liu W.Q., Generalized KarhunenLoève
transform, IEEE Signal Processing Letters, 1998, V.5, 141143.
 Vapnik V., Statistical Learning Theory, Wiley, 1998.
 Ocaña F.A., Aguilera A.M., Valderrama M.J., Functional principal
componenets analysis by choice of norm, J. Multivariate Anal., 1999, V.71, 262276.
 Tipping M.E., Bishop C.M., Probabilistic principal component analysis,
J. of the Royal Statistical Society, Ser. A, 1999, V.61, 611619.
 Tipping M.E., Bishop C.M., Mixtures of probabilistic principal component analysers,
Neural Computation, 1999, V.11, 443482.
 Schölkopf B., Smola A.J., Müller K.R., Kernel principal component analysis,
in Advances in Kernel Methods. Support Vector Learning, Editors B. Schölkopf, C.J.C. Burges and A.J. Smola,
Cambridge, MIT Press, 1999, 327352.
 Tenenbaum J.B., de Silva V., Langford J.C., A global geometric framework for
nonlinear dimensionality reduction, Science, 2000, V.290, Issue 5500, 23192323.
 Rowers S.T., Saul L.K., Nonlinear dimensionality reduction by locally linear embedding,
Science, 2000, V.290, Issue 5500, 23232326.
 Cristianini N., ShaweTaylor J., An introduction to support vector machines and other
kernelbased learning methods, Cambridge, Cambridge University Press, 2000.
 Yamada I., Sekiguchi T., Sakaniwa K., Reduced rank Volterra
filter for robust identification of nonlinear systems,
in Proc. 2nd Int. Workshop on Multidimensional (ND)
Systems  NDS2000, Poland, Czocha Castle, 2000, 171175.
 Hua Y., Nikpour M., Stoica P., Optimal reducedrank estimation and filtering,
IEEE Trans. on Signal Processing, 2001, V.49, 457469.
 Kneip A., Utikal K.J., Inference for density families using
functional principal component analysis, Journal of the American Statistical Association,
2001, V.96, 519542.
 Honig M.L., Xiao W., Performance of reducedrank linear interferrence suppression,
IEEE Trans. on Information Theory, 2001, V.47, 19281946.
 Chen W., Mitra U., Schniter P., On the equivalence of three
rediced rank linear estimators with applications to DSCDMA,
IEEE Trans. on Information Theory, 2002, V.48, 26092614.
 Honig M.L., Goldstein J.S., Adaptive reducedrank
interference suppression based on multistage Wiener filter, IEEE Trans. on Communications,
2002, V.50, 986994.
 Stock J.H., Watson M.W., Forecasting using principal components from a large number of predictors,
Journal of the American Statistical Association, 2002, V.97, 11671179.
 Fukunaga K., Introduction to statistical pattern recognition,
Boston, Academic Press, 1990.
 Kraut S., Anderson R.H., Krolik J.L., A generalized KarhunenLoève
basis for efficient estimation of tropospheric refractivity using radar clutter, IEEE Trans.
on Signal Processing, 2004, V.52, 4860.
 Torokhti A., Howlett P., An optimal filter of the second order,
IEEE Trans. on Signal Processing, 2001, V.49, 10441048.
 Torokhti A., Howlett P., Optimal fixed rank
transform of the second degree, IEEE Trans. on Circuits and
Systems. Part II, Analog & Digital Signal Processing, 2001, V.48, 309315.
 Torokhti A., Howlett P., Pearce C., New perspectives on optimal
transforms of random vectors, Optimization: Theory and Applications, to appear.
 Torokhti A., Howlett P., Constructing fixed rank optimal estimators with method of recurrent
best approximations, J. Multivariate Analysis, 2002, V.86, 293309.
 Torokhti A., Howlett P., Best operator approximation in modelling of nonlinear Systems,
IEEE Trans. on Circuits and
Systems. Part I, Fundamental Theory and Applications, 2002, V.49, 17921798.
 Torokhti A., Howlett P., Method of recurrent best estimators of second degree for optimal
filtering of random signals, Signal Processing, 2003, V.83, 10131024.
 Torokhti A., Howlett P., Best causal mathematical models for a nonlinear system,
IEEE Trans. on Circuits and
Systems. Part I, Fundamental Theory and Applications, to appear.
 Sontag E.D., Polynomial response maps,
Lecture Notes in Control and Information Sciences, 1979. Vol. 13.
 Chen S., Billings S.A., Representation of nonlinear systems: NARMAX model,
Int. J. Control, 1989, V.49, 10131032.
 Howlett P.G., Torokhti A.P., Pearce C.E.M., A
philosophy for the modelling of realistic nonlinear systems, Proc. of Amer.
Math. Soc., 2003, V.132, 353363.
 Cotlar M., Cignoli R., An introduction to functional analysis,
Amsterdam  London, NorthHolland Publishing Company, 1974, 114116.
 Perlovsky L.I., Marzetta T.L., Estimating a covariance matrix from incomplete realizations
of a random vector, IEEE Trans. on Signal Processing, 1992, V.40, 20972100.
 Kauermann G., Carroll R.J., A note on the efficiency of Sandwich covariance matrix estimation,
Journal of the American Statistical Association, 2001, V.96, 13871396.
 Schneider M.K., Willsky A.S., A Krylov subspace method for covariance approximation
and simulation of a random process and fields, Int. J. Multidim. Syst. & Signal Processing,
2003, V.14, 295318.
 Kubokawa T., Srivastava M.S., Estimating the covariance matrix: a new approach,
J. Multivariate Analysis, 2003, V.86, 2847.
 Ledoit O., Wolf M., A wellconditioned estimator for largedimensional covariance matrices,
J. Multivariate Analysis, 2004, V.88, 365411.
 Leung P.L., Ng F.Y., Improved estimation of a covariance matrix
in an elliptically contoured matrix distribution, J. Multivariate Analysis, 2004,
V.88, 131137.
 Higham N.J., Stable iterations for the matrix square root, Numerical Algorithms,
1997, V.15, 227241.
 Golub G.H., van Loan C.F., Matrix computations, Baltimore, Johns Hopkins University Press, 1996.
 Kowalski M.A., Sikorski K.A., Stenger F., Selected topics in approximation and computations,
New York  Oxford, Oxford University Press, 1995.
 BenIsrael A., Greville T.N.E., Generalized inverses: theory and applications, New York,
John Wiley & Sons, 1974.
 Mathews V.J., Sicuranza G.L., Polynomial signal processing, J. Wiley & Sons, 2001.
 Goldstein J.S., Reed I., Scharf L.L., A multistage representation of the
Wiener filter based on orthogonal projections, IEEE Trans. on Information Theory, 1998,
V.44, 29432959.

