1932

Abstract

The brain's function is to enable adaptive behavior in the world. To this end, the brain processes information about the world. The concept of representation links the information processed by the brain back to the world and enables us to understand what the brain does at a functional level. The appeal of making the connection between brain activity and what it represents has been irresistible to neuroscience, despite the fact that representational interpretations pose several challenges: We must define which aspects of brain activity matter, how the code works, and how it supports computations that contribute to adaptive behavior. It has been suggested that we might drop representational language altogether and seek to understand the brain, more simply, as a dynamical system. In this review, we argue that the concept of representation provides a useful link between dynamics and computational function and ask which aspects of brain activity should be analyzed to achieve a representational understanding. We peel the onion of brain representations in search of the layers (the aspects of brain activity) that matter to computation. The article provides an introduction to the motivation and mathematics of representational models, a critical discussion of their assumptions and limitations, and a preview of future directions in this area.

Loading

Article metrics loading...

/content/journals/10.1146/annurev-neuro-080317-061906
2019-07-08
2024-03-28
Loading full text...

Full text loading...

/deliver/fulltext/neuro/42/1/annurev-neuro-080317-061906.html?itemId=/content/journals/10.1146/annurev-neuro-080317-061906&mimeType=html&fmt=ahah

Literature Cited

  1. Abbott LF, Dayan P 1999. The effect of correlated variability on the accuracy of a population code. Neural Comput. 11:91–101
    [Google Scholar]
  2. Afraz S-R, Kiani R, Esteky H 2006. Microstimulation of inferotemporal cortex influences face categorization. Nature 442:692–95
    [Google Scholar]
  3. Allefeld C, Haynes JD 2014. Searchlight-based multi-voxel pattern analysis of fMRI by cross-validated MANOVA. NeuroImage 89:345–57
    [Google Scholar]
  4. Averbeck BB, Latham PE, Pouget A 2006. Neural correlations, population coding and computation. Nat. Rev. Neurosci. 7:535866
    [Google Scholar]
  5. Bargmann CI, Marder E 2013. From the connectome to brain function. Nat. Methods 10:648390
    [Google Scholar]
  6. Bechtel W 1998. Representations and cognitive explanations: assessing the dynamicist's challenge in cognitive science. Cogn. Sci. 22:3295318
    [Google Scholar]
  7. Brentano F 1874. Psychology from an Empirical Standpoint Abingdon, UK: Routledge
  8. Cai MB, Schuck NW, Pillow J, Niv Y 2016. A Bayesian method for reducing bias in neural representational similarity analysis. Advances in Neural Information Processing Systems DD Lee, M Sugiyama, UV Luxburg, I Guyon, R Garnett495260 Cambridge, MA: MIT Press
    [Google Scholar]
  9. Carlson T, Tovar DA, Alink A, Kriegeskorte N 2013. Representational dynamics of object vision: the first 1000 ms. J. Vis. 13:101
    [Google Scholar]
  10. Carlson TA, Schrater P, He SY 2003. Patterns of activity in the categorical representation of objects. J. Cogn. Neurosci. 15:70417
    [Google Scholar]
  11. Chen BL, Hall DH, Chklovskii DB 2006. Wiring optimization can relate neuronal structure and function. PNAS 103:12472328
    [Google Scholar]
  12. Chklovskii DB, Koulakov AA 2004. Maps in the brain: What can we learn from them. Annu. Rev. Neurosci. 27:36992
    [Google Scholar]
  13. Churchland MM, Cunningham JP, Kaufman MT, Foster JD, Nuyujukian P et al. 2012. Neural population dynamics during reaching. Nature 487:5156
    [Google Scholar]
  14. Cichy RM, Pantazis D, Oliva A 2014. Resolving human object recognition in space and time. Nat. Neurosci. 17:345562
    [Google Scholar]
  15. Cox DD, Savoy RL 2003. Functional magnetic resonance imaging (fMRI) brain reading: detecting and classifying distributed patterns of fMRI activity in human visual cortex. NeuroImage 19:26170
    [Google Scholar]
  16. Cunningham JP, Yu BM 2014. Dimensionality reduction for large-scale neural recordings. Nat. Neurosci. 17:1115009
    [Google Scholar]
  17. deCharms RC, Zador A 2000. Neural representation and the cortical code. Annu. Rev. Neurosci. 23:61347
    [Google Scholar]
  18. Dennett D 1987. The Intentional Stance Cambridge, MA: MIT Press
  19. DiCarlo JJ, Cox D 2007. Untangling invariant object recognition. Trends Cogn. Sci. 11:33441
    [Google Scholar]
  20. DiCarlo JJ, Zoccolan D, Rust NC 2012. How does the brain solve visual object recognition. Neuron 73:341534
    [Google Scholar]
  21. Diedrichsen J 2019. Representational models and the feature fallacy. The Cognitive Neurosciences MS Gazzaniga, GR Mangun, D Poeppel Cambridge, MA: MIT Press In press
    [Google Scholar]
  22. Diedrichsen J, Kriegeskorte N 2017. Representational models: a common framework for understanding encoding, pattern-component, and representational-similarity analysis. PLOS Comput. Biol. 13:4e1005508
    [Google Scholar]
  23. Diedrichsen J, Provost S, Hossein ZH 2016. On the distribution of cross-validated Mahalanobis distances. arxiv:1607.01371 [stat.AP]
    [Google Scholar]
  24. Diedrichsen J, Ridgway GR, Friston KJ, Wiestler T 2011. Comparing the similarity and spatial structure of neural representations: a pattern-component model. NeuroImage 55:4166578
    [Google Scholar]
  25. Diedrichsen J, Yokoi A, Arbuckle SA 2018. Pattern component modeling: a flexible approach for understanding the representational structure of brain activity patterns. NeuroImage 180:11933
    [Google Scholar]
  26. Dumoulin SO, Wandell BA 2008. Population receptive field estimates in human visual cortex. NeuroImage 39:264760
    [Google Scholar]
  27. Edelman S 1998. Representation is representation of similarities. Behav. Brain Sci. 21:444967
    [Google Scholar]
  28. Edelman S, Grill-Spector K, Kushnir T, Malach R 1998. Toward direct visualization of the internal shape representation space by fMRI. Psychobiology 26:430921
    [Google Scholar]
  29. Ejaz N, Hamada M, Diedrichsen J 2015. Hand use predicts the structure of representations in sensorimotor cortex. Nat. Neurosci. 18:103440
    [Google Scholar]
  30. Felleman DJ, Van Essen DC 1991. Distributed hierarchical processing in the primate cerebral cortex. Cereb. Cortex 1:1147
    [Google Scholar]
  31. Formisano E, Kim DS, Di Salle F, van de Moortele PF, Ugurbil K, Goebel R 2003. Mirror-symmetric tonotopic maps in human primary auditory cortex. Neuron 40:485969
    [Google Scholar]
  32. Ganguli S, Sompolinsky H 2012. Compressed sensing, sparsity, and dimensionality in neuronal information processing and data analysis. Annu. Rev. Neurosci. 35:485508
    [Google Scholar]
  33. Gao P, Ganguli S 2015. On simplicity and complexity in the brave new world of large-scale neuroscience. Curr. Opin. Neurobiol. 32:14855
    [Google Scholar]
  34. Grill-Spector K, Henson R, Martin A 2006. Repetition and the brain: neural models of stimulus-specific effects. Trends Cogn. Sci. 10:11423
    [Google Scholar]
  35. Güçlü U, van Gerven MA 2015. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. J. Neurosci. 35:271000514
    [Google Scholar]
  36. Haxby JV, Connolly AC, Guntupalli JS 2014. Decoding neural representational spaces using multivariate pattern analysis. Annu. Rev. Neurosci. 37:43556
    [Google Scholar]
  37. Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P 2001. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293:5539242530
    [Google Scholar]
  38. Haynes J-D 2015. A primer on pattern-based approaches to fMRI: principles, pitfalls, and perspectives. Neuron 87:225770
    [Google Scholar]
  39. Haynes J-D, Rees G 2006. Decoding mental states from brain activity in humans. Nat. Rev. Neurosci. 7:752334
    [Google Scholar]
  40. Hebart MN, Baker CI 2017. Deconstructing multivariate decoding for the study of brain function. NeuroImage 180:418
    [Google Scholar]
  41. Hong H, Yamins DL, Majaj NJ, DiCarlo JJ 2016. Explicit information for category-orthogonal object properties increases along the ventral stream. Nat. Neurosci. 19:461322
    [Google Scholar]
  42. Hung CP, Kreiman G, Poggio T, DiCarlo JJ 2005. Fast readout of object identity from macaque inferior temporal cortex. Science 310:574986366
    [Google Scholar]
  43. Huth AG, de Heer WA, Griffiths TL, Theunissen FE, Gallant JL 2016. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature 532:45358
    [Google Scholar]
  44. Huth AG, Nishimoto S, Vu AT, Gallant JL 2012. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron 76:121024
    [Google Scholar]
  45. Ibsen H 1867. Peer Gynt: A Dramatic Poem. Copenhagen: Gyldendal
  46. Kamitani Y, Tong F 2005. Decoding the visual and subjective contents of the human brain. Nat. Neurosci. 8:567985
    [Google Scholar]
  47. Kay KN, Naselaris T, Prenger RJ, Gallant JL 2008. Identifying natural images from human brain activity. Nature 452:718535255
    [Google Scholar]
  48. King JR, Dehaene S 2014. Characterizing the dynamics of mental representations: the temporal generalization method. Trends Cogn. Sci. 18:420310
    [Google Scholar]
  49. Kobak D, Brendel W, Constantinidis C, Feierstein CE, Kepecs A et al. 2016. Demixed principal component analysis of neural population data. eLife 5:e10989
    [Google Scholar]
  50. Kriegeskorte N 2011. Pattern-information analysis: from stimulus decoding to computational-model testing. NeuroImage 56:241121
    [Google Scholar]
  51. Kriegeskorte N 2015. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annu. Rev. Vis. Sci. 1:41746
    [Google Scholar]
  52. Kriegeskorte N, Bandettini P 2007. Analyzing for information, not activation, to exploit high-resolution fMRI. NeuroImage 38:464962
    [Google Scholar]
  53. Kriegeskorte N, Diedrichsen J 2016. Inferring brain-computational mechanisms with models of activity measurements. Philos. Trans. R. Soc. B 371:170520160278
    [Google Scholar]
  54. Kriegeskorte N, Douglas PK 2018a. Cognitive computational neuroscience. Nat. Neurosci. 29:114860
    [Google Scholar]
  55. Kriegeskorte N, Douglas PK 2018b. Interpreting encoding and decoding models. arXiv:1812.00278 [q-bio.NC]
    [Google Scholar]
  56. Kriegeskorte N, Formisano E, Sorger B, Goebel R 2007. Individual faces elicit distinct response patterns in human anterior temporal cortex. PNAS 104:51206005
    [Google Scholar]
  57. Kriegeskorte N, Goebel R, Bandettini P 2006. Information-based functional brain mapping. PNAS 103:10386368
    [Google Scholar]
  58. Kriegeskorte N, Kievit RA 2013. Representational geometry: integrating cognition, computation, and the brain. Trends Cogn. Sci. 17:40112
    [Google Scholar]
  59. Kriegeskorte N, Mur M, Bandettini P 2008a. Representational similarity analysis: connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2:4
    [Google Scholar]
  60. Kriegeskorte N, Mur M, Ruff DA, Kiani R, Bodurka J et al. 2008b. Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron 60:112641
    [Google Scholar]
  61. Millikan RG 1989. Biosemantics. J. Philos. 86:628197
    [Google Scholar]
  62. Misaki M, Kim Y, Bandettini PA, Kriegeskorte N 2010. Comparison of multivariate classifiers and response normalizations for pattern-information fMRI. NeuroImage 53:110318
    [Google Scholar]
  63. Mitchell TM, Shinkareva SV, Carlson A, Chang KM, Malave VL et al. 2008. Predicting human brain activity associated with the meanings of nouns. Science 320:119195
    [Google Scholar]
  64. Moreno-Bote R, Beck J, Kanitscheider I, Pitkow X, Latham P, Pouget A 2014. Information-limiting correlations. Nat. Neurosci. 17:10141017
    [Google Scholar]
  65. Mur M, Bandettini PA, Kriegeskorte N 2009. Revealing representational content with pattern-information fMRI: an introductory guide. Soc. Cogn. Affect. Neurosci. 4:11019
    [Google Scholar]
  66. Naselaris T, Kay KN 2015. Resolving ambiguities of MVPA using explicit models of representation. Trends Cogn. Sci. 19:1055154
    [Google Scholar]
  67. Naselaris T, Kay KN, Nishimoto S, Gallant JL 2011. Encoding and decoding in fMRI. NeuroImage 56:240010
    [Google Scholar]
  68. Nili H, Wingfield C, Walther A, Su L, Marslen-Wilson W, Kriegeskorte N 2014. A toolbox for representational similarity analysis. PLOS Comput. Biol. 10:e1003553
    [Google Scholar]
  69. Norman KA, Polyn SM, Detre GJ, Haxby JV 2006. Beyond mindreading: multi-voxel pattern analysis of fMRI data. Trends Cogn. Sci. 10:42430
    [Google Scholar]
  70. Norman-Haignere S, Kanwisher NG, McDermott JH 2015. Distinct cortical pathways for music and speech revealed by hypothesis-free voxel decomposition. Neuron 88:128196
    [Google Scholar]
  71. Olshausen B, Field D 2004. Sparse coding of sensory inputs. Curr. Opin. Neurobiol. 14:48187
    [Google Scholar]
  72. Paninski L, Cunningham J 2017. Neural data science: accelerating the experiment-analysis-theory cycle in large-scale neuroscience. bioRxiv 196949
    [Google Scholar]
  73. Paninski L, Pillow J, Lewi J 2007. Statistical models for neural encoding, decoding, and optimal stimulus design. Prog. Brain Res. 165:493507
    [Google Scholar]
  74. Parvizi J, Jacques C, Foster BL, Withoft N, Rangarajan V et al. 2012. Electrical stimulation of human fusiform face-selective regions distorts face perception. J. Neurosci. 32:1491520
    [Google Scholar]
  75. Pereira F, Mitchell T, Botvinick M 2009. Machine learning classifiers and fMRI: a tutorial overview. NeuroImage 45:1 Suppl.S199209
    [Google Scholar]
  76. Poeppel D 2012. The maps problem and the mapping problem: two challenges for a cognitive neuroscience of speech and language. Cogn. Neuropsychol. 29:1–23455
    [Google Scholar]
  77. Rieke F, Warland D, de Ruyter van Steveninck R, Bialek W 1999. Spikes: Exploring the Neural Code Cambridge, MA: MIT Press
  78. Salzman CD, Britten KH, Newsome WT 1990. Cortical microstimulation influences perceptual judgements of motion direction. Nature 346:628017477
    [Google Scholar]
  79. Shea N 2018. Representation in Cognitive Science Oxford, UK: Oxford Univ. Press
  80. Shenoy KV, Sahani M, Churchland MM 2013. Cortical control of arm movements: a dynamical systems perspective. Annu. Rev. Neurosci. 36:33759
    [Google Scholar]
  81. Simoncelli EP, Olshausen BA 2001. Natural image statistics and neural representation. Annu. Rev. Neurosci. 24:1193216
    [Google Scholar]
  82. Tong F, Pratte MS 2012. Decoding patterns of human brain activity. Annu. Rev. Psychol. 63:483509
    [Google Scholar]
  83. Tootell RB, Hadjikhani NK, Mendola JD, Marrett S, Dale AM 1998. From retinotopy to recognition: fMRI in human visual cortex. Trends Cogn. Sci. 2:517483
    [Google Scholar]
  84. Van Gelder T 1998. The dynamical hypothesis in cognitive science. Behav. Brain Sci. 21:561528
    [Google Scholar]
  85. van Gerven MA 2017. A primer on encoding models in sensory neuroscience. J. Math. Psychol. 76:17283
    [Google Scholar]
  86. Varoquaux G, Raamana PR, Engemann DA, Hoyos-Idrobo A, Schwartz Y, Thirion B 2017. Assessing and tuning brain decoders: cross-validation, caveats, and guidelines. NeuroImage 145:16679
    [Google Scholar]
  87. Walther A, Nili H, Ejaz N, Alink A, Kriegeskorte N, Diedrichsen J 2016. Reliability of dissimilarity measures for multi-voxel pattern analysis. NeuroImage 137:188200
    [Google Scholar]
  88. Wilson SP, Bednar JA 2015. What, if anything, are topological maps for. Dev. Neurobiol. 75:666781
    [Google Scholar]
  89. Wu MCK, David SV, Gallant JL 2006. Complete functional characterization of sensory neurons by system identification. Annu. Rev. Neurosci. 29:477505
    [Google Scholar]
  90. Yamins DLK, DiCarlo JJ 2016. Using goal-driven deep learning models to understand sensory cortex. Nat. Neurosci. 19:35665
    [Google Scholar]
  91. Yu BM, Cunningham JP, Santhanam G, Ryu SI, Shenoy KV, Sahani M 2009. Gaussian-process factor analysis for low-dimensional single-trial analysis of neural population activity. J. Neurophysiol. 102:161435
    [Google Scholar]
/content/journals/10.1146/annurev-neuro-080317-061906
Loading
/content/journals/10.1146/annurev-neuro-080317-061906
Loading

Data & Media loading...

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error