1932

Abstract

Large amounts of multidimensional data represented by multiway arrays or tensors are prevalent in modern applications across various fields such as chemometrics, genomics, physics, psychology, and signal processing. The structural complexity of such data provides vast new opportunities for modeling and analysis, but efficiently extracting information content from them, both statistically and computationally, presents unique and fundamental challenges. Addressing these challenges requires an interdisciplinary approach that brings together tools and insights from statistics, optimization, and numerical linear algebra, among other fields. Despite these hurdles, significant progress has been made in the past decade. This review seeks to examine some of the key advancements and identify common threads among them, under a number of different statistical settings.

Loading

Article metrics loading...

/content/journals/10.1146/annurev-statistics-112723-034548
2025-03-07
2025-06-20
Loading full text...

Full text loading...

/deliver/fulltext/statistics/12/1/annurev-statistics-112723-034548.html?itemId=/content/journals/10.1146/annurev-statistics-112723-034548&mimeType=html&fmt=ahah

Literature Cited

  1. Acar E, Dunlavy DM, Kolda TG. 2011.. A scalable optimization approach for fitting canonical tensor decompositions. . J. Chemom. 25:(2):6786
    [Crossref] [Google Scholar]
  2. Allen G. 2012.. Sparse higher-order principal components analysis. . Proc. Mach. Learn. Res. 22::2736
    [Google Scholar]
  3. Anandkumar A, Ge R, Hsu D, Kakade SM, Telgarsky M. 2014a.. Tensor decompositions for learning latent variable models. . J. Mach. Learn. Res. 15::2773832
    [Google Scholar]
  4. Anandkumar A, Ge R, Janzamin M. 2014b.. Guaranteed non-orthogonal tensor decomposition via alternating rank-1 updates. . arXiv:1402.5180 [cs.LG]
  5. Anandkumar A, Ge R, Janzamin M. 2015.. Learning overcomplete latent variable models through tensor methods. . Proc. Mach. Learn. Res. 40::36112
    [Google Scholar]
  6. Arous GB, Mei S, Montanari A, Nica M. 2019.. The landscape of the spiked tensor model. . Commun. Pure Appl. Math. 72:(11):2282330
    [Crossref] [Google Scholar]
  7. Auddy A, Yuan M. 2022.. On estimating rank-one spiked tensors in the presence of heavy tailed errors. . IEEE Trans. Inform. Theory 68:(12):805375
    [Crossref] [Google Scholar]
  8. Auddy A, Yuan M. 2023a.. Large dimensional independent component analysis: statistical optimality and computational tractability. . arXiv:2303.18156 [math.ST]
  9. Auddy A, Yuan M. 2023b.. Perturbation bounds for (nearly) orthogonally decomposable tensors with statistical applications. . Inform. Inference 12:(2):104472
    [Crossref] [Google Scholar]
  10. Bahadori MT, Yu QR, Liu Y. 2014.. Fast multivariate spatio-temporal analysis via low rank tensor learning. . In NIPS'14: Proceedings of the 27th International Conference on Neural Information Processing Systems, ed. Z Ghahramani, M Welling, C Cortes, ND Lawrence, KQ Weinberger , pp. 349199. Red Hook, NY:: Curran
    [Google Scholar]
  11. Barak B, Moitra A. 2016.. Noisy tensor completion via the sum-of-squares hierarchy. . Proc. Mach. Learn. Res. 49::41745
    [Google Scholar]
  12. Beckmann CF, DeLuca M, Devlin JT, Smith SM. 2005.. Investigations into resting-state connectivity using independent component analysis. . Philos. Trans. R. Soc. B: Biol. Sci. 360:(1457):100113
    [Crossref] [Google Scholar]
  13. Belkin M, Rademacher L, Voss J. 2013.. Blind signal separation in the presence of Gaussian noise. . Proc. Mach. Learn. Res. 30::27087
    [Google Scholar]
  14. Belkin M, Rademacher L, Voss J. 2018.. Eigenvectors of orthogonally decomposable functions. . SIAM J. Comput. 47:(2):547615
    [Crossref] [Google Scholar]
  15. Benson AR, Gleich DF, Leskovec J. 2016.. Higher-order organization of complex networks. . Science 353:(6295):16366
    [Crossref] [Google Scholar]
  16. Bhatia R. 2013.. Matrix Analysis. Grad. Texts Math. Vol. 169. New York:: Springer Sci. Bus. Media
    [Google Scholar]
  17. Bi X, Tang X, Yuan Y, Zhang Y, Qu A. 2021.. Tensors in statistics. . Annu. Rev. Stat. Appl. 8::34568
    [Crossref] [Google Scholar]
  18. Cai C, Li G, Poor HV, Chen Y. 2022a.. Nonconvex low-rank tensor completion from noisy data. . Oper. Res. 70:(2):121937
    [Crossref] [Google Scholar]
  19. Cai C, Poor HV, Chen Y. 2020.. Uncertainty quantification for nonconvex tensor completion: confidence intervals, heteroscedasticity and optimality. . Proc. Mach. Learn. Res. 119::127182
    [Google Scholar]
  20. Cai JF, Li J, Xia D. 2022b.. Generalized low-rank plus sparse tensor estimation by fast Riemannian optimization. . J. Am. Stat. Assoc. 118::2588604
    [Crossref] [Google Scholar]
  21. Cai JF, Li J, Xia D. 2022c.. Provable tensor-train format tensor completion by Riemannian optimization. . J. Mach. Learn. Res. 23:(123):177
    [Google Scholar]
  22. Carroll JD, Chang JJ. 1970.. Analysis of individual differences in multidimensional scaling via an n-way generalization of Eckart-Young decomposition. . Psychometrika 35:(3):283319
    [Crossref] [Google Scholar]
  23. Chen H, Raskutti G, Yuan M. 2019.. Non-convex projected gradient descent for generalized low-rank tensor regression. . J. Mach. Learn. Res. 20:(1):172208
    [Google Scholar]
  24. Cichocki A, Mandic D, De Lathauwer L, Zhou G, Zhao Q, et al. 2015.. Tensor decompositions for signal processing applications: from two-way to multiway component analysis. . IEEE Signal Proc. Mag. 32:(2):14563
    [Crossref] [Google Scholar]
  25. Comon P. 1994.. Independent component analysis, a new concept?. Signal Proc. 36:(3):287314
    [Crossref] [Google Scholar]
  26. Comon P, Jutten C. 2010.. Handbook of Blind Source Separation: Independent Component Analysis and Applications. New York:: Academic
    [Google Scholar]
  27. Cong F, Lin QH, Kuang LD, Gong XF, Astikainen P, Ristaniemi T. 2015.. Tensor decomposition of EEG signals: a brief review. . J. Neurosci. Methods 248::5969
    [Crossref] [Google Scholar]
  28. Coppi R, Bolasco S. 1989.. Multiway Data Analysis. Amsterdam:: North-Holland
    [Google Scholar]
  29. De Lathauwer L, De Moor B, Vandewalle J. 2000a.. A multilinear singular value decomposition. . SIAM J. Matrix Anal. Appl. 21:(4):125378
    [Crossref] [Google Scholar]
  30. De Lathauwer L, De Moor B, Vandewalle J. 2000b.. On the best rank-1 and rank-(R1, R2,…,RN) approximation of higher-order tensors. . SIAM J. Matrix Anal. Appl. 21:(4):132442
    [Crossref] [Google Scholar]
  31. Edelman A, Arias TA, Smith ST. 1998.. The geometry of algorithms with orthogonality constraints. . SIAM J. Matrix Anal. Appl. 20:(2):30353
    [Crossref] [Google Scholar]
  32. Gandy S, Recht B, Yamada I. 2011.. Tensor completion and low-n-rank tensor recovery via convex optimization. . Inverse Probl. 27:(2):025010
    [Crossref] [Google Scholar]
  33. Ge R, Huang F, Jin C, Yuan Y. 2015.. Escaping from saddle points—online stochastic gradient for tensor decomposition. . Proc. Mach. Learn. Res. 40::797842
    [Google Scholar]
  34. Ge Z, Song Z. 2007.. Process monitoring based on independent component analysis–principal component analysis (ICA–PCA) and similarity factors. . Ind. Eng. Chem. Res. 46:(7):205463
    [Crossref] [Google Scholar]
  35. Ghadermarzy N, Plan Y, Yilmaz Ö. 2019.. Near-optimal sample complexity for convex tensor completion. . Inform. Inference 8:(3):577619
    [Crossref] [Google Scholar]
  36. Ghoshdastidar D, Dukkipati A. 2014.. Consistency of spectral partitioning of uniform hypergraphs under planted partition model. . In NIPS'14: Proceedings of the 27th International Conference on Neural Information Processing Systems, ed. Z Ghahramani, M Welling, C Cortes, ND Lawrence, KQ Weinberger , pp. 397405. Red Hook, NY:: Curran
    [Google Scholar]
  37. Hackbusch W. 2012.. Tensor Spaces and Numerical Tensor Calculus. New York:: Springer
    [Google Scholar]
  38. Han R, Luo Y, Wang M, Zhang AR. 2022a.. Exact clustering in tensor block model: statistical optimality and computational limit. . J. R. Stat. Soc. Ser. B 84:(5):166698
    [Crossref] [Google Scholar]
  39. Han R, Willett R, Zhang AR. 2022b.. An optimal statistical and computational framework for generalized tensor estimation. . Ann. Stat. 50:(1):129
    [Google Scholar]
  40. Harshman R. 1970.. Foundations of the PARAFAC procedure: model and conditions for an explanatory factor analysis. Tech. Rep., UCLA Work. Pap. Phon. 16 , Univ. Calif., Los Angeles:
    [Google Scholar]
  41. Hillar CJ, Lim LH. 2013.. Most tensor problems are NP-hard. . J. ACM 60:(6):45
    [Crossref] [Google Scholar]
  42. Hitchcock FL. 1927.. The expression of a tensor or a polyadic as a sum of products. . J. Math. Phys. 6:(1–4):16489
    [Crossref] [Google Scholar]
  43. Hopkins SB, Shi J, Steurer D. 2015.. Tensor principal component analysis via sum-of-square proofs. . Proc. Mach. Learn. Res. 40::9561006
    [Google Scholar]
  44. Hunyadi B, Dupont P, Van Paesschen W, Van Huffel S. 2017.. Tensor decompositions and data fusion in epileptic electroencephalography and functional magnetic resonance imaging data. . Wiley Interdiscip. Rev. Data Min. Knowledge Discov. 7:(1):e1197
    [Crossref] [Google Scholar]
  45. Hyvarinen A. 1999.. Fast and robust fixed-point algorithms for independent component analysis. . IEEE Trans. Neural Netw. 10:(3):62634
    [Crossref] [Google Scholar]
  46. Hyvarinen A, Karhunen J, Oja E. 2002.. Independent component analysis. . Stud. Inform. Control 11:(2):2057
    [Google Scholar]
  47. Hyvärinen A, Oja E. 2000.. Independent component analysis: algorithms and applications. . Neural Netw. 13:(4–5):41130
    [Crossref] [Google Scholar]
  48. Jain P, Oh S. 2014.. Provable tensor factorization with missing data. . In NIPS'14: Proceedings of the 27th International Conference on Neural Information Processing Systems, ed. Z Ghahramani, M Welling, C Cortes, ND Lawrence, KQ Weinberger , pp. 143139. Red Hook, NY:: Curran
    [Google Scholar]
  49. Jain P, Tewari A, Kar P. 2014.. On iterative hard thresholding methods for high-dimensional m-estimation. . In NIPS'14: Proceedings of the 27th International Conference on Neural Information Processing Systems, ed. Z Ghahramani, M Welling, C Cortes, ND Lawrence, KQ Weinberger , pp. 68593. Red Hook, NY:: Curran
    [Google Scholar]
  50. James CJ, Hesse CW. 2004.. Independent component analysis for biomedical signals. . Physiol. Meas. 26:(1):R15
    [Crossref] [Google Scholar]
  51. Janzamin M, Ge R, Kossaifi J, Anandkumar A. 2019.. Spectral learning on matrices and tensors. . Found. Trends Mach. Learn. 12:(5–6):393536
    [Crossref] [Google Scholar]
  52. Jiang B, Ma S, Zhang S. 2015.. Tensor principal component analysis via convex optimization. . Math. Program. 150:(2):42357
    [Crossref] [Google Scholar]
  53. Jing BY, Li T, Lyu Z, Xia D. 2021.. Community detection on mixture multilayer networks via regularized tensor decomposition. . Ann. Stat. 49:(6):3181205
    [Crossref] [Google Scholar]
  54. Jolliffe IT. 2002.. Principal Component Analysis for Special Types of Data. New York:: Springer
    [Google Scholar]
  55. Kang HJ, Kawasawa YI, Cheng F, Zhu Y, Xu X, et al. 2011.. Spatio-temporal transcriptome of the human brain. . Nature 478:(7370):48389
    [Crossref] [Google Scholar]
  56. Ke ZT, Shi F, Xia D. 2019.. Community detection for hypergraph networks via regularized tensor power iteration. . arXiv:1909.06503 [stat.ME]
  57. Kivelä M, Arenas A, Barthelemy M, Gleeson JP, Moreno Y, Porter MA. 2014.. Multilayer networks. . J. Complex Netw. 2:(3):20371
    [Crossref] [Google Scholar]
  58. Kressner D, Steinlechner M, Vandereycken B. 2014.. Low-rank tensor completion by Riemannian optimization. . BIT Numer. Math. 54::44768
    [Crossref] [Google Scholar]
  59. Kroonenberg PM. 2008.. Applied Multiway Data Analysis. New York:: John Wiley & Sons
    [Google Scholar]
  60. Kruskal JB. 1977.. Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics. . Linear Algebra Appl. 18:(2):95138
    [Crossref] [Google Scholar]
  61. Lee JM, Yoo C, Lee IB. 2004.. Statistical monitoring of dynamic processes based on dynamic independent component analysis. . Chem. Eng. Sci. 59:(14):29953006
    [Crossref] [Google Scholar]
  62. Liu J, Musialski P, Wonka P, Ye J. 2012.. Tensor completion for estimating missing values in visual data. . IEEE Trans. Pattern Anal. Mach. Intell. 35:(1):20820
    [Crossref] [Google Scholar]
  63. Liu T, Yuan M, Zhao H. 2022.. Characterizing spatiotemporal transcriptome of the human brain via low-rank tensor decomposition. . Stat. Biosci. 14::485513
    [Crossref] [Google Scholar]
  64. Luo Y, Raskutti G, Yuan M, Zhang AR. 2021.. A sharp blockwise tensor perturbation bound for orthogonal iteration. . J Mach. Learn. Res. 22:(1):810653
    [Google Scholar]
  65. Lyu Z, Li T, Xia D. 2023a.. Optimal clustering of discrete mixtures: binomial, Poisson, block models, and multi-layer networks. . arXiv:2311.15598 [math.ST]
  66. Lyu Z, Xia D, Zhang Y. 2023b.. Latent space model for higher-order networks and generalized tensor decomposition. . J. Comput. Graph. Stat. 32::132036
    [Crossref] [Google Scholar]
  67. Ma T, Shi J, Steurer D. 2016.. Polynomial-time tensor decompositions with sum-of-squares. . In 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS), pp. 43846. Piscataway, NJ:: IEEE
    [Google Scholar]
  68. McCullagh P. 2018.. Tensor Methods in Statistics. Mineola, NY:: Courier Dover
    [Google Scholar]
  69. Montanari A, Sun N. 2018.. Spectral algorithms for tensor completion. . Commun. Pure Appl. Math. 71:(11):2381425
    [Crossref] [Google Scholar]
  70. Mu C, Hsu D, Goldfarb D. 2015.. Successive rank-one approximations for nearly orthogonally decomposable symmetric tensors. . SIAM J. Matrix Anal. Appl. 36:(4):163859
    [Crossref] [Google Scholar]
  71. Mu C, Hsu D, Goldfarb D. 2017.. Greedy approaches to symmetric orthogonal tensor decomposition. . SIAM J. Matrix Anal. Appl. 38:(4):121026
    [Crossref] [Google Scholar]
  72. Mu C, Huang B, Wright J, Goldfarb D. 2014.. Square deal: lower bounds and improved relaxations for tensor recovery. . Proc. Mach. Learn. Res. 32:(2):7381
    [Google Scholar]
  73. Mucha PJ, Richardson T, Macon K, Porter MA, Onnela JP. 2010.. Community structure in time-dependent, multiscale, and multiplex networks. . Science 328:(5980):87678
    [Crossref] [Google Scholar]
  74. Ollila E. 2009.. The deflation-based FastICA estimator: statistical analysis revisited. . IEEE Trans. Signal Proc. 58:(3):152741
    [Crossref] [Google Scholar]
  75. Onton J, Westerfield M, Townsend J, Makeig S. 2006.. Imaging human EEG dynamics using independent component analysis. . Neurosci. Biobehav. Rev. 30:(6):80822
    [Crossref] [Google Scholar]
  76. Ouyang J, Yuan M. 2023.. On the multiway principal component analysis. . arXiv:2302.07216 [stat.ME]
  77. Potechin A, Steurer D. 2017.. Exact tensor completion with sum-of-squares. . Proc. Mach. Learn. Res. 65::161973
    [Google Scholar]
  78. Raskutti G, Yuan M, Chen H. 2019.. Convex regularization for high-dimensional multiresponse tensor regression. . Ann. Stat. 47:(3):155484
    [Crossref] [Google Scholar]
  79. Richard E, Montanari A. 2014.. A statistical model for tensor PCA. . In NIPS'14: Proceedings of the 27th International Conference on Neural Information Processing Systems, ed. Z Ghahramani, M Welling, C Cortes, ND Lawrence, KQ Weinberger , pp. 2897905. Red Hook, NY:: Curran
    [Google Scholar]
  80. Romera-Paredes B, Aung H, Bianchi-Berthouze N, Pontil M. 2013.. Multilinear multitask learning. . Proc. Mach. Learn. Res. 28:(3):144452
    [Google Scholar]
  81. Sanchez E, Kowalski BR. 1990.. Tensorial resolution: a direct trilinear decomposition. . J. Chemom. 4:(1):2945
    [Crossref] [Google Scholar]
  82. Schramm T, Steurer D. 2017.. Fast and robust tensor decomposition with applications to dictionary learning. . Proc. Mach. Learn. Res. 65::176093
    [Google Scholar]
  83. Sharan V, Valiant G. 2017.. Orthogonalized ALS: a theoretically principled tensor decomposition algorithm for practical use. . Proc. Mach. Learn. Res. 70::3095104
    [Google Scholar]
  84. Sidiropoulos ND, De Lathauwer L, Fu X, Huang K, Papalexakis EE, Faloutsos C. 2017.. Tensor decomposition for signal processing and machine learning. . IEEE Trans. Signal Proc. 65:(13):355182
    [Crossref] [Google Scholar]
  85. Signoretto M, Van de Plas R, De Moor B, Suykens JA. 2011.. Tensor versus matrix completion: a comparison with application to spectral data. . IEEE Signal Proc. Lett. 18:(7):4036
    [Crossref] [Google Scholar]
  86. Song Q, Ge H, Caverlee J, Hu X. 2019.. Tensor completion algorithms in big data analytics. . ACM Trans. Knowledge Discov. Data 13:(1):6
    [Crossref] [Google Scholar]
  87. Stewart GW, Sun J. 1990.. Matrix Perturbation Theory. New York:: Academic
    [Google Scholar]
  88. Sun WW, Li L. 2017.. Store: sparse tensor response regression and neuroimaging analysis. . J. Mach. Learn. Res. 18:(135):137
    [Google Scholar]
  89. Sun WW, Lu J, Liu H, Cheng G. 2017.. Provable sparse tensor decomposition. . J. R. Stat. Soc. Ser. B 79:(3):899916
    [Crossref] [Google Scholar]
  90. Tang R, Yuan M, Zhang AR. 2023.. Mode-wise principal subspace pursuit and matrix spiked covariance model. . arXiv:2307.00575 [stat.ME]
  91. Tomioka R, Suzuki T. 2013.. Convex tensor decomposition via structured Schatten norm regularization. . In NIPS'13: Proceedings of the 26th International Conference on Neural Information Processing Systems, ed. CJC Burges, L Bottou, M Welling, Z Ghahramani, KQ Weinberger , pp. 133139. Red Hook, NY:: Curran
    [Google Scholar]
  92. Tucker LR. 1963.. Implications of factor analysis of three-way matrices for measurement of change. . Probl. Meas. Change 15::12237
    [Google Scholar]
  93. Tucker LR. 1966.. Some mathematical notes on three-mode factor analysis. . Psychometrika 31:(3):279311
    [Crossref] [Google Scholar]
  94. Wang M, Zeng Y. 2019.. Multiway clustering via tensor block models. . In Proceedings of the 33rd International Conference on Neural Information Processing Systems, ed. HM Wallach, H Larochelle, A Beygelzimer, F d'Alché-Buc, EB Fox , pp. 71525. Red Hook, NY:: Curran
    [Google Scholar]
  95. Xia D, Yuan M. 2019.. On polynomial time methods for exact low-rank tensor completion. . Found. Comput. Math. 19:(6):1265313
    [Crossref] [Google Scholar]
  96. Xia D, Yuan M. 2021.. Effective tensor sketching via sparsification. . IEEE Trans. Inform. Theory 67:(2):135669
    [Crossref] [Google Scholar]
  97. Xia D, Yuan M, Zhang CH. 2021.. Statistically optimal and computationally efficient low rank tensor completion from noisy entries. . Ann. Stat. 49:(1):7699
    [Crossref] [Google Scholar]
  98. Xia D, Zhou F. 2019.. The sup-norm perturbation of HOSVD and low rank tensor denoising. . J. Mach. Learn. Res. 20:(1):220647
    [Google Scholar]
  99. Yu R, Cheng D, Liu Y. 2015.. Accelerated online low rank tensor learning for multivariate spatiotemporal streams. . Proc. Mach. Learn. Res. 37::23847
    [Google Scholar]
  100. Yu R, Liu Y. 2016.. Learning from multiway data: simple and efficient tensor regression. . Proc. Mach. Learn. Res. 48::37381
    [Google Scholar]
  101. Yuan M, Zhang CH. 2016.. On tensor completion via nuclear norm minimization. . Found. Comput. Math. 16:(4):103168
    [Crossref] [Google Scholar]
  102. Yuan M, Zhang CH. 2017.. Incoherent tensor norms and their applications in higher order tensor completion. . IEEE Trans. Inform. Theory 63:(10):675366
    [Crossref] [Google Scholar]
  103. Zhang A, Han R. 2019.. Optimal sparse singular value decomposition for high-dimensional high-order data. . J. Am. Stat. Assoc. 114:(528):170825
    [Crossref] [Google Scholar]
  104. Zhang A, Xia D. 2018.. Tensor SVD: statistical and computational limits. . IEEE Trans. Inform. Theory 64:(11):731138
    [Crossref] [Google Scholar]
  105. Zhang AR, Luo Y, Raskutti G, Yuan M. 2020.. ISLET: fast and optimal low-rank tensor regression via importance sketching. . SIAM J. Math. Data Sci. 2:(2):44479
    [Crossref] [Google Scholar]
  106. Zhou H, Li L, Zhu H. 2013.. Tensor regression with applications in neuroimaging data analysis. . J. Am. Stat. Assoc. 108:(502):54052
    [Crossref] [Google Scholar]
  107. Zhou Y, Chen Y. 2023.. Heteroskedastic tensor clustering. . arXiv:2311.02306 [math.ST]
/content/journals/10.1146/annurev-statistics-112723-034548
Loading
/content/journals/10.1146/annurev-statistics-112723-034548
Loading

Data & Media loading...

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error