Neurons in early visual cortical areas not only represent incoming visual information but are also engaged by higher level cognitive processes, including attention, working memory, imagery, and decision-making. Are these cognitive effects an epiphenomenon or are they functionally relevant for these mental operations? We review evidence supporting the hypothesis that the modulation of activity in early visual areas has a causal role in cognition. The modulatory influences allow the early visual cortex to act as a multiscale cognitive blackboard for read and write operations by higher visual areas, which can thereby efficiently exchange information. This blackboard architecture explains how the activity of neurons in the early visual cortex contributes to scene segmentation and working memory, and relates to the subject's inferences about the visual world. The architecture also has distinct advantages for the processing of visual routines that rely on a number of sequentially executed processing steps.


Article metrics loading...

Loading full text...

Full text loading...


Literature Cited

  1. Albers AM, Kok P, Toni I, Dijkerman HC, de Lange FP. 2013. Shared representations for working memory and mental imagery in early visual cortex. Curr. Biol. 23:1427–31 [Google Scholar]
  2. Albright TD. 2012. On the perception of probable things: neural substrates of associative memory, imagery, and perception. Neuron 74:227–45 [Google Scholar]
  3. Armstrong KM, Fitzgerald JK, Moore T. 2006. Changes in visual receptive fields with microstimulation of frontal cortex. Neuron 50:791–98 [Google Scholar]
  4. Baruni JK, Lau B, Salzman CD. 2015. Reward expectation differentially modulates attentional behavior and activity in visual area V4. Nat. Neurosci. 18:1656–63 [Google Scholar]
  5. Bhatt R, Carpenter GA, Grossberg S. 2007. Texture segregation by visual cortex: perceptual grouping, attention, and learning. Vis. Res. 47:3173–211 [Google Scholar]
  6. Britten KH, Newsome WT, Shadlen MN, Celebrini S, Movshon JA. 1996. A relationship between behavioral choice and the visual responses of neurons in macaque MT. Vis. Neurosci. 13:87–100 [Google Scholar]
  7. Bullier J. 2001. Integrated model of visual processing. Brain Res. Brain Res. Rev. 36:96–107 [Google Scholar]
  8. Burt PJ, Adelson EH. 1983. The Laplacian pyramid as a compact image code. IEEE Trans. Commun. 31:532–40 [Google Scholar]
  9. Cavanagh P. 2011. Visual cognition. Vis. Res. 51:1538–51 [Google Scholar]
  10. Chelazzi L, Miller EK, Duncan J, Desimone R. 2001. Responses of neurons in macaque area V4 during memory-guided visual search. Cereb. Cortex 11:761–72 [Google Scholar]
  11. Chen M, Yan Y, Gong X, Gilbert CD, Liang H, Li W. 2014. Incremental integration of global contours through interplay between visual cortical areas. Neuron 82:682–94 [Google Scholar]
  12. Cohen MR, Newsome WT. 2008. Context-dependent changes in functional circuitry in visual area MT. Neuron 60:1162–73 [Google Scholar]
  13. Duncan J, Humphreys GK, Ward R. 1997. Competitive brain activity in visual attention. Curr. Opin. Neurobiol. 7:255–61 [Google Scholar]
  14. Ekstrom LB, Roelfsema PR, Arsenault JT, Bonmassar G, Vanduffel W. 2008. Bottom-up dependent gating of frontal signals in early visual cortex. Science 321:414–17 [Google Scholar]
  15. Felleman DJ, Van Essen DC. 1991. Distributed hierarchical processing in the primate cerebral cortex. Cereb. Cortex 1:1–47 [Google Scholar]
  16. Ferrera VP, Rudolph KK, Maunsell JH. 1994. Responses of neurons in the parietal and temporal visual pathways during a motion task. J. Neurosci. 14:6171–86 [Google Scholar]
  17. Gilad A, Meirovithz E, Slovin H. 2013. Population responses to contour integration: early encoding of discrete elements and late perceptual grouping. Neuron 78:389–402 [Google Scholar]
  18. Gold JI, Shadlen MN. 2001. Neural computations that underlie decisions about sensory stimuli. Trends Cogn. Sci. 5:10–16 [Google Scholar]
  19. Gold JI, Shadlen MN. 2007. The neural basis of decision making. Annu. Rev. Neurosci. 30:535–74 [Google Scholar]
  20. Goldman-Rakic PS. 1995. Cellular basis of working memory. Neuron 14:477–85 [Google Scholar]
  21. Güçlü U, van Gerven MAJ. 2015. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. J. Neurosci. 35:10005–14 [Google Scholar]
  22. Haefner RM, Berkes P, Fiser J. 2015. The implications of perception as probabilistic inference for correlated neural variability during behavior. ArXiv:1409.0257 [q-bio.NC] http://arxiv.org/pdf/1409.0257.pdf
  23. Hamker FH. 2005. The reentry hypothesis: the putative interaction of the frontal eye field, ventrolateral prefrontal cortex, and areas V4, IT for attention and eye movement. Cereb. Cortex 15:431–47 [Google Scholar]
  24. Harrison SA, Tong F. 2009. Decoding reveals the contents of visual working memory in early visual areas. Nature 485:632–35 [Google Scholar]
  25. Heinen K, Jolij J, Lamme VAF. 2005. Figure–ground segregation requires two distinct periods of activity in V1: a transcranial magnetic stimulation study. NeuroReport 16:1483–87 [Google Scholar]
  26. Hochstein S, Ahissar M. 2002. View from the top: hierarchies and reverse hierarchies in the visual system. Neuron 36:791–804 [Google Scholar]
  27. Houtkamp R, Roelfsema PR. 2010. Parallel and serial grouping of image elements in visual perception. J. Exp. Psychol. 36:1443–59 [Google Scholar]
  28. Houtkamp R, Spekreijse H, Roelfsema PR. 2003. A gradual spread of attention during mental curve tracing. Percept. Psychophys. 65:1136–44 [Google Scholar]
  29. Hung CP, Kreiman G, Poggio T, DiCarlo JJ. 2005. Fast readout of object identity from macaque inferior temporal cortex. Science 310:863–66 [Google Scholar]
  30. Itti L, Koch C. 2001. Computational modelling of visual attention. Nat. Rev. Neurosci. 2:194–203 [Google Scholar]
  31. Jolicoeur P, Ingleton M. 1991. Size invariance in curve tracing. Mem. Cogn. 19:21–36 [Google Scholar]
  32. Jolicoeur P, Ullman S, MacKay M. 1991. Visual curve tracing properties. J. Exp. Psychol. 17:997–1022 [Google Scholar]
  33. Juan C-H, Walsh V. 2003. Feedback to V1: a reverse hierarchy in vision. Exp. Brain Res. 150:259–63 [Google Scholar]
  34. Kaas JH. 2008. The evolution of the complex sensory and motor systems of the human brain. Brain Res. Bull. 75:384–90 [Google Scholar]
  35. Khayat PS, Spekreijse H, Roelfsema PR. 2004. Correlates of transsaccadic integration in the primary visual cortex of the monkey. PNAS 101:12712–17 [Google Scholar]
  36. Koivisto M, Railo H, Revonsuo A, Vanni S, Salminen-Vaparanta N. 2011. Recurrent processing in V1/V2 contributes to categorization of natural scenes. J. Neurosci. 31:2488–92 [Google Scholar]
  37. Kok P, Brouwer GJ, van Gerven MAJ, de Lange FP. 2013. Prior expectations bias sensory representations in visual cortex. J. Neurosci. 33:16275–84 [Google Scholar]
  38. Kok P, Failing MF, de Lange FP. 2014. Prior expectations evoke stimulus templates in the primary visual cortex. J. Cogn. Neurosci. 26:1546–54 [Google Scholar]
  39. Kok P, Jehee JFM, de Lange FP. 2012. Less is more: Expectation sharpens representations in the primary visual cortex. Neuron 75:265–70 [Google Scholar]
  40. Kosslyn SM. 1996. Image and Brain Cambridge, MA: MIT Press [Google Scholar]
  41. Kosslyn SM, Ganis G, Thompson WL. 2001. Neural foundations of imagery. Nat. Rev. Neurosci. 2:635–42 [Google Scholar]
  42. Kosslyn SM, Pascual-Leone A, Felician O, Camposano S, Keenan JP. et al. 1999. The role of area 17 in visual imagery: convergent evidence from PET and rTMS. Science 284:167–70 [Google Scholar]
  43. Lamme VAF. 1995. The neurophysiology of figure–ground segregation in primary visual cortex. J. Neurosci. 15:1605–15 [Google Scholar]
  44. Law C-T, Gold JI. 2008. Neural correlates of perceptual learning in a sensory-motor, but not a sensory, cortical area. Nat. Neurosci. 11:505–13 [Google Scholar]
  45. LeCun Y, Bengio Y, Hinton G. 2015. Deep learning. Nature 521:436–44 [Google Scholar]
  46. Lee H, Simpson GV, Logothetis NK, Rainer G. 2005. Phase locking of single neuron activity to theta oscillations during working memory in monkey extrastriate visual cortex. Neuron 45:147–56 [Google Scholar]
  47. Lee TS, Yang CF, Romero RD, Mumford D. 2002. Neural activity in early visual cortex reflects behavioral experience and higher-order perceptual saliency. Nat. Neurosci. 5:589–97 [Google Scholar]
  48. Li W, Piëch V, Gilbert CD. 2006. Contour saliency in primary visual cortex. Neuron 50:951–62 [Google Scholar]
  49. Logothetis NK, Wandell BA. 2004. Interpreting the BOLD Signal. Annu. Rev. Physiol. 66:735–69 [Google Scholar]
  50. Long J, Shelhamer E, Darrell T. 2014. Fully convolutional networks for semantic segmentation. ArXiv:1411.4038 [cs.CV]
  51. Lorteije JAM, Zylberberg A, Ouellette BG, De Zeeuw CI, Sigman M, Roelfsema PR. 2015. The formation of hierarchical decisions in the visual cortex. Neuron 87:1344–56 [Google Scholar]
  52. Markov NT, Misery P, Falchier A, Lamy C, Vezoli J. et al. 2011. Weight consistency specifies regularities of macaque cortical networks. Cereb. Cortex 21:1254–72 [Google Scholar]
  53. Melcher D, Colby CL. 2008. Trans-saccadic perception. Trends Cogn. Sci. 12:466–73 [Google Scholar]
  54. Mendoza-Halliday D, Torres S, Martinez-Trujillo JC. 2014. Sharp emergence of feature-selective sustained activity along the dorsal visual pathway. Nat. Neurosci. 17:1255–62 [Google Scholar]
  55. Mesulam M. 2008. Representation, inference, and transcendent encoding in neurocognitive networks of the human brain. Ann. Neurol. 64:367–78 [Google Scholar]
  56. Moore T, Armstrong KM. 2003. Selective gating of visual signals by microstimulation of frontal cortex. Nature 421:370–73 [Google Scholar]
  57. Moro SI, Tolboom M, Khayat PS, Roelfsema PR. 2010. Neuronal activity in the visual cortex reveals the temporal order of cognitive operations. J. Neurosci. 30:16293–303 [Google Scholar]
  58. Muckli L, Petro LS. 2013. Network interactions: non-geniculate input to V1. Curr. Opin. Neurobiol. 23:195–201 [Google Scholar]
  59. Mumford D. 1991. On the computational architecture of the neocortex. I. The role of the thalamo-cortical loop. Biol. Cybern. 65:135–45 [Google Scholar]
  60. Mumford D, Kosslyn SM, Hillger LA, Herrnstein RJ. 1987. Discriminating figure from ground: the role of edge detection and region growing. PNAS 84:7354–58 [Google Scholar]
  61. Nienborg H, Cumming BG. 2009. Decision-related activity in sensory neurons reflects more than a neuron's causal effect. Nature 459:89–92 [Google Scholar]
  62. Nienborg H, Roelfsema PR. 2015. Belief states as a framework to explain extra-retinal influences in visual cortex. Curr. Opin. Neurobiol. 32:45–52 [Google Scholar]
  63. Pasternak T, Greenlee MW. 2005. Working memory in primate sensory systems. Nat. Rev. Neurosci. 6:97–107 [Google Scholar]
  64. Pearson J, Naselaris T, Holmes EA, Kosslyn SM. 2015. Mental imagery: functional mechanisms and clinical applications. Trends Cogn. Sci. 19:590–602 [Google Scholar]
  65. Platt ML, Glimcher PW. 1999. Neural correlates of decision variables in parietal cortex. Nature 400:233–38 [Google Scholar]
  66. Pooresmaeili A, Poort J, Roelfsema PR. 2014. Simultaneous selection by object-based attention in visual and frontal cortex. PNAS 111:6467–72 [Google Scholar]
  67. Pooresmaeili A, Poort J, Thiele A, Roelfsema PR. 2010. Separable codes for attention and luminance contrast in the primary visual cortex. J. Neurosci. 30:12701–11 [Google Scholar]
  68. Pooresmaeili A, Roelfsema PR. 2014. A growth-cone model for the spread of object-based attention during contour grouping. Curr. Biol. 24:2869–77 [Google Scholar]
  69. Poort J, Raudies F, Wannig A, Lamme VAF, Neumann H, Roelfsema PR. 2012. The role of attention in figure–ground segregation in areas V1 and V4 of the visual cortex. Neuron 75:143–56 [Google Scholar]
  70. Rainer G, Rao SC, Miller EK. 1999. Prospective coding for objects in primate prefrontal cortex. J. Neurosci. 19:5493–505 [Google Scholar]
  71. Rock I, Victor J. 1964. Vision and touch: an experimentally created conflict between the two senses. Science 143:594–96 [Google Scholar]
  72. Rockland KS, Virga A. 1989. Terminal arbors of individual “feedback” axons projecting from area V2 to V1 in the macaque monkey: a study using immunohistochochemistry of anterogradely transported Phaseolus vulgaris-leucoagglutinin. J. Comp. Neurol. 285:54–72 [Google Scholar]
  73. Roelfsema PR. 2005. Elemental operations in vision. Trends Cogn. Sci. 9:226–33 [Google Scholar]
  74. Roelfsema PR. 2006. Cortical algorithms for perceptual grouping. Annu. Rev. Neurosci. 29:203–27 [Google Scholar]
  75. Roelfsema PR, Houtkamp R. 2011. Incremental grouping of image elements in vision. Atten. Percept. Psychophys. 73:2542–72 [Google Scholar]
  76. Roelfsema PR, Khayat PS, Spekreijse H. 2003. Subtask sequencing in the primary visual cortex. PNAS 100:5467–72 [Google Scholar]
  77. Roelfsema PR, Lamme VAF, Spekreijse H. 1998. Object-based attention in the primary visual cortex of the macaque monkey. Nature 395:376–81 [Google Scholar]
  78. Roelfsema PR, Lamme VAF, Spekreijse H, Bosch H. 2002. Figure–ground segregation in a recurrent network architecture. J. Cogn. Neurosci. 14:525–37 [Google Scholar]
  79. Roelfsema PR, Spekreijse H. 2001. The representation of erroneously perceived stimuli in the primary visual cortex. Neuron 31:853–63 [Google Scholar]
  80. Sakai K, Miyashita Y. 1991. Neural organization for the long-term memory of paired associates. Nature 354:152–55 [Google Scholar]
  81. Schall JD. 2001. Neural basis of deciding, choosing and acting. Nat. Rev. Neurosci. 2:33–42 [Google Scholar]
  82. Schlack A, Albright TD. 2007. Remembering visual motion: neural correlates of associative plasticity and motion recall in cortical area MT. Neuron 53:881–90 [Google Scholar]
  83. Self MW, Kooijmans RN, Supèr H, Lamme VAF, Roelfsema PR. 2012. Different glutamate receptors convey feedforward and recurrent processing in macaque V1. PNAS 109:11031–36 [Google Scholar]
  84. Self MW, van Kerkoerle T, Supèr H, Roelfsema PR. 2013. Distinct roles of the cortical layers of area V1 in figure–ground segregation. Curr. Biol. 23:2121–29 [Google Scholar]
  85. Serences JT, Ester EF, Vogel EK, Awh E. 2009. Stimulus-specific delay activity in human primary visual cortex. Psychol. Sci. 20:207–14 [Google Scholar]
  86. Shadlen MN, Britten KH, Newsome WT, Movshon JA. 1996. A computational analysis of the relationship between neuronal and behavioral responses to visual motion. J. Neurosci. 16:1486–1510 [Google Scholar]
  87. Shadlen MN, Newsome WT. 1996. Motion perception: seeing and deciding. PNAS 93:628–33 [Google Scholar]
  88. Shepard RN, Cooper LA. 1982. Mental Images and Their Transformations Cambridge, MA: MIT Press [Google Scholar]
  89. Shepard RN, Metzler J. 1971. Mental rotation of three-dimensional objects. Science 171:701–3 [Google Scholar]
  90. Spaak E, Fonken Y, Jensen O, de Lange FP. 2016. The neural mechanisms of prediction in visual search. Cereb. Cortex. In press doi: 10.1093/cercor/bhv210 [Google Scholar]
  91. Stănişor L, van der Togt C, Pennartz CMA, Roelfsema PR. 2013. A unified selection signal for attention and reward in primary visual cortex. PNAS 110:9136–41 [Google Scholar]
  92. Stokes M, Thompson R, Cusack R, Duncan J. 2009. Top-down activation of shape-specific population codes in visual cortex during mental imagery. J. Neurosci. 29:1565–72 [Google Scholar]
  93. Summerfield C, de Lange FP. 2014. Expectation in perceptual decision making: neural and computational mechanisms. Nat. Rev. Neurosci. 15:745–56 [Google Scholar]
  94. Supèr H, Spekreijse H, Lamme VA. 2001. A neural correlate of working memory in the monkey primary visual cortex. Science 293:120–24 [Google Scholar]
  95. Thorpe S, Fize D, Marlot C. 1996. Speed of processing in the human visual system. Nature 381:520–22 [Google Scholar]
  96. Ullman S. 1984. Visual routines. Cognition 18:97–159 [Google Scholar]
  97. van de Ven V, Jacobs C, Sack AT. 2012. Topographic contribution of early visual cortex to short-term memory consolidation: a transcranial magnetic stimulation study. J. Neurosci. 32:4–11 [Google Scholar]
  98. van der Velde F, de Kamps M. 2001. From knowing what to knowing where: modeling object-based attention with feedback disinhibition of activation. J. Cogn. Neurosci. 13:479–91 [Google Scholar]
  99. van der Velde F, de Kamps M. 2006. Neural blackboard architectures of combinatorial structures in cognition. Behav. Brain Sci. 29:37–70 [Google Scholar]
  100. van Kerkoerle T, Self MW, Roelfsema PR. 2014. Effects of attention and working memory in the different layers of monkey primary visual cortex. Soc. Neurosci. Abstr. 263:13 [Google Scholar]
  101. Wannig A, Stanisor L, Roelfsema PR. 2011. Automatic spread of attentional response modulation along Gestalt criteria in primary visual cortex. Nat. Neurosci. 14:1243–44 [Google Scholar]
  102. Wimmer K, Compte A, Roxin A, Peixoto D, Renart A, de la Rocha J. 2015. Sensory integration dynamics in a hierarchical network explains choice probabilities in cortical area MT. Nat. Commun. 6:6177 [Google Scholar]
  103. Wolfson SS, Landy MS. 1998. Examining edge- and region-based texture analysis mechanisms. Vis. Res. 38:439–46 [Google Scholar]
  104. Yamins DLK, Hong H, Cadieu C, Solomon EA, Seibert D, DiCarlo JJ. 2014. Performance-optimized hierarchical models predict neural responses in higher visual cortex. PNAS 111:8619–24 [Google Scholar]
  105. Zaksas D, Pasternak T. 2006. Directional signals in the prefrontal cortex and in area MT during a working memory for visual motion task. J. Neurosci. 26:11726–42 [Google Scholar]
  106. Zipser K, Lamme VAF, Schiller PH. 1996. Contextual modulation in primary visual cortex. J. Neurosci. 16:7376–89 [Google Scholar]
  107. Zylberberg A, Dehaene S, Roelfsema PR, Sigman M. 2011. The human Turing machine: a neural framework for mental programs. Trends Cogn. Sci. 15:293–300 [Google Scholar]

Data & Media loading...

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error