1932
0
  1. Home
  2. A-Z Publications
  3. Annual Review of Vision Science
  4. Volume 7, 2021
  5. Article

Annual Review of Vision Science

Volume 7, 2021

Visual Remapping

Abstract

Our visual system is fundamentally retinotopic. When viewing a stable scene, each eye movement shifts object features and locations on the retina. Thus, sensory representations must be updated, or remapped, across saccades to align presaccadic and postsaccadic inputs. The earliest remapping studies focused on anticipatory, presaccadic shifts of neuronal spatial receptive fields. Over time, it has become clear that there are multiple forms of remapping and that different forms of remapping may be mediated by different neural mechanisms. This review attempts to organize the various forms of remapping into a functional taxonomy based on experimental data and ongoing debates about forward versus convergent remapping, presaccadic versus postsaccadic remapping, and spatial versus attentional remapping. We integrate findings from primate neurophysiological, human neuroimaging and behavioral, and computational modeling studies. We conclude by discussing persistent open questions related to remapping, with specific attention to binding of spatial and featural information during remapping and speculations about remapping's functional significance.

Keyword(s): attention, eye movements, retinotopic, saccade, spatiotopic, visual perception
Loading

Article metrics loading...

/content/journals/10.1146/annurev-vision-032321-100012
2021-09-15
2024-04-20
Loading full text...

Full text loading...

/deliver/fulltext/vision/7/1/annurev-vision-032321-100012.html?itemId=/content/journals/10.1146/annurev-vision-032321-100012&mimeType=html&fmt=ahah

Literature Cited

  1. Andersen RA, Mountcastle VB. 1983. The influence of the angle of gaze upon the excitability of the light-sensitive neurons of the posterior parietal cortex. J. Neurosci. 3:3532–48
     [Google Scholar]
  2. Anderson C, Van Essen DC 1987. Shifter circuits: a computational strategy for dynamic aspects of visual processing. PNAS 84:6297–301
     [Google Scholar]
  3. Arkesteijn K, Belopolsky AV, Smeets JBJ, Donk M. 2019. The limits of predictive remapping of attention across eye movements. Front. Psychol. 10:1146
     [Google Scholar]
  4. Baltaretu BR, Monaco S, Velji-Ibrahim J, Luabeya GN, Crawford JD. 2020. Parietal cortex integrates saccade and object orientation signals to update grasp plans. J. Neurosci. 40:234525–35
     [Google Scholar]
  5. Bergelt J, Hamker FH. 2019. Spatial updating of attention across eye movements: a neuro-computational approach. J. Vis. 19:710
     [Google Scholar]
  6. Casarotti M, Lisi M, Umiltà C, Zorzi M. 2012. Paying attention through eye movements: a computational investigation of the premotor theory of spatial attention. J. Cogn. Neurosci. 24:71519–31
     [Google Scholar]
  7. Cavanagh P, Hunt AR, Afraz A, Rolfs M. 2010. Visual stability based on remapping of attention pointers. Trends Cogn. Sci. 14:4147–53
     [Google Scholar]
  8. Cha O, Chong SC. 2014. The background is remapped across saccades. Exp. Brain Res. 232:2609–18
     [Google Scholar]
  9. Churan J, Guitton D, Pack CC. 2011. Context dependence of receptive field remapping in superior colliculus. J. Neurophysiol. 106:41862–74
     [Google Scholar]
  10. Churan J, Guitton D, Pack CC. 2012. Perisaccadic remapping and rescaling of visual responses in macaque superior colliculus. PLOS ONE 7:12e52195
     [Google Scholar]
  11. Connor CE, Gallant JL, Preddie DC, Van Essen DC. 1996. Responses in area V4 depend on the spatial relationship between stimulus and attention. J. Neurophysiol. 75:31306–8
     [Google Scholar]
  12. Dowd EW, Golomb JD. 2019. Object-feature binding survives dynamic shifts of spatial attention. Psychol. Sci. 30:3343–61
     [Google Scholar]
  13. Dowd EW, Golomb JD. 2020. The binding problem after an eye movement. Atten. Percept. Psychophys. 82:1168–80
     [Google Scholar]
  14. Duhamel JR, Colby CL, Goldberg ME. 1992. The updating of the representation of visual space in parietal cortex by intended eye movements. Science 255:504090–92
     [Google Scholar]
  15. Dunkley BT, Baltaretu B, Crawford JD. 2016. Trans-saccadic interactions in human parietal and occipital cortex during the retention and comparison of object orientation. Cortex 82:263–76
     [Google Scholar]
  16. Edwards G, VanRullen R, Cavanagh P. 2018. Decoding trans-saccadic memory. J. Neurosci. 38:51114–23
     [Google Scholar]
  17. Fabius JH, Fracasso A, Acunzo DJ, Van der Stigchel S, Melcher D. 2020. Low-level visual information is maintained across saccades, allowing for a postsaccadic hand-off between visual areas. J. Neurosci. 40:499476–86
     [Google Scholar]
  18. Fabius JH, Fracasso A, Van der Stigchel S. 2016. Spatiotopic updating facilitates perception immediately after saccades. Sci. Rep. 6:34488
     [Google Scholar]
  19. Fairhall SL, Schwarzbach J, Lingnau A, Van Koningsbruggen MG, Melcher D. 2017. Spatiotopic updating across saccades revealed by spatially-specific fMRI adaptation. NeuroImage 147:339–45
     [Google Scholar]
  20. Fischer B, Boch R. 1981a. Enhanced activation of neurons in prelunate cortex before visually guided saccades of trained rhesus monkeys. Exp. Brain Res. 44:2129–37
     [Google Scholar]
  21. Fischer B, Boch R. 1981b. Selection of visual targets activates prelunate cortical cells in trained rhesus monkey. Exp. Brain Res. 41:3–4431–33
     [Google Scholar]
  22. Ganmor E, Landy MS, Simoncelli EP. 2015. Near-optimal integration of orientation information across saccades. J. Vis. 15:168
     [Google Scholar]
  23. Golomb JD. 2019. Remapping locations and features across saccades: a dual-spotlight theory of attentional updating. Curr. Opin. Psychol. 29:211–18
     [Google Scholar]
  24. Golomb JD, Chun MM, Mazer JA. 2008. The native coordinate system of spatial attention is retinotopic. J. Neurosci. 28:4210654–62
     [Google Scholar]
  25. Golomb JD, Kanwisher N 2012. Retinotopic memory is more precise than spatiotopic memory. PNAS 109:51796–801
     [Google Scholar]
  26. Golomb JD, Kupitz CN, Thiemann CT. 2014a. The influence of object location on identity: a “spatial congruency bias. .” J. Exp. Psychol. Gen. 143:62262–78
     [Google Scholar]
  27. Golomb JD, L'Heureux ZE, Kanwisher N 2014b. Feature-binding errors after eye movements and shifts of attention. Psychol. Sci. 25:51067–78
     [Google Scholar]
  28. Golomb JD, Marino AC, Chun MM, Mazer JA. 2011. Attention doesn't slide: Spatiotopic updating after eye movements instantiates a new, discrete attentional locus. Atten. Percept. Psychophys. 73:17–14
     [Google Scholar]
  29. Golomb JD, Nguyen-Phuc AY, Mazer JA, McCarthy G, Chun MM. 2010a. Attentional facilitation throughout human visual cortex lingers in retinotopic coordinates after eye movements. J. Neurosci. 30:3110493–506
     [Google Scholar]
  30. Golomb JD, Pulido VZ, Albrecht AR, Chun MM, Mazer JA. 2010b. Robustness of the retinotopic attentional trace after eye movements. J. Vis. 10:319
     [Google Scholar]
  31. Gottlieb JP, Kusunoki M, Goldberg ME. 1998. The representation of visual salience in monkey parietal cortex. Nature 391:6666481–84
     [Google Scholar]
  32. Haider B, Krause MR, Duque A, Yu Y, Touryan J et al. 2010. Synaptic and network mechanisms of sparse and reliable visual cortical activity during nonclassical receptive field stimulation. Neuron 65:1107–21
     [Google Scholar]
  33. Hall NJ, Colby CL. 2011. Remapping for visual stability. Philos. Trans. R. Soc. B 366: 1564.528–39
     [Google Scholar]
  34. Harrison WJ, Bex PJ. 2014. Integrating retinotopic features in spatiotopic coordinates. J. Neurosci. 34:217351–60
     [Google Scholar]
  35. Hartmann TS, Zirnsak M, Marquis M, Hamker FH, Moore T. 2017. Two types of receptive field dynamics in area V4 at the time of eye movements?. Front. Syst. Neurosci. 11:13
     [Google Scholar]
  36. Hayhoe M, Lachter J, Feldman J. 1991. Integration of form across saccadic eye movements. Perception 20:3393–402
     [Google Scholar]
  37. He D, Mo C, Fang F. 2017. Predictive feature remapping before saccadic eye movements. J. Vis. 17:514
     [Google Scholar]
  38. Holcombe AO 2009. The binding problem. Encyclopedia of Perception EB Goldstein 205–8 Thousand Oaks, CA: SAGE
     [Google Scholar]
  39. Hunt AR, Cavanagh P. 2011. Remapped visual masking. J. Vis. 11:113
     [Google Scholar]
  40. Irwin DE, Yantis S, Jonides J. 1983. Evidence against visual integration across saccadic eye movements. Percept. Psychophys. 34:149–57
     [Google Scholar]
  41. Joiner WM, Cavanaugh J, Wurtz RH. 2011. Modulation of shifting receptive field activity in frontal eye field by visual salience. J. Neurophysiol. 106:31179–90
     [Google Scholar]
  42. Jonikaitis D, Szinte M, Rolfs M, Cavanagh P. 2013. Allocation of attention across saccades. J. Neurophysiol. 109:51425–34
     [Google Scholar]
  43. Khayat PS, Spekreijse H, Roelfsema PR. 2006. Attention lights up new object representations before the old ones fade away. J. Neurosci. 26:1138–42
     [Google Scholar]
  44. Knapen T, Rolfs M, Wexler M, Cavanagh P. 2010. The reference frame of the tilt aftereffect. J. Vis. 10:18
     [Google Scholar]
  45. Knapen T, Swisher JD, Tong F, Cavanagh P. 2016. Oculomotor remapping of visual information to foveal retinotopic cortex. Front. Syst. Neurosci. 10:54
     [Google Scholar]
  46. Kowler E, Anderson E, Dosher B, Blaser E 1995. The role of attention in the programming of saccades. Vis. Res. 35:131897–916
     [Google Scholar]
  47. Lee J, Maunsell JH. 2009. A normalization model of attentional modulation of single unit responses. PLOS ONE 4:2e4651
     [Google Scholar]
  48. Lescroart MD, Kanwisher N, Golomb JD. 2016. No evidence for automatic remapping of stimulus features or location found with fMRI. Front. Syst. Neurosci. 10:53
     [Google Scholar]
  49. Marino AC, Mazer JA. 2016. Perisaccadic updating of visual representations and attentional states: linking behavior and neurophysiology. Front. Syst. Neurosci. 10:3
     [Google Scholar]
  50. Marino AC, Mazer JA. 2018. Saccades trigger predictive updating of attentional topography in area V4. Neuron 98:2429–38.e4
     [Google Scholar]
  51. Mathôt S, Theeuwes J. 2010a. Evidence for the predictive remapping of visual attention. Exp. Brain Res. 200:1117–22
     [Google Scholar]
  52. Mathôt S, Theeuwes J. 2010b. Gradual remapping results in early retinotopic and late spatiotopic inhibition of return. Psychol. Sci. 21:121793–98
     [Google Scholar]
  53. Mathôt S, Theeuwes J. 2013. A reinvestigation of the reference frame of the tilt-adaptation aftereffect. Sci. Rep. 3:1152
     [Google Scholar]
  54. Mayo JP, Sommer MA. 2010. Shifting attention to neurons. Trends Cogn. Sci. 14:9389
     [Google Scholar]
  55. McConkie GW, Currie CB. 1996. Visual stability across saccades while viewing complex pictures. J. Exp. Psychol. Hum. Percept. Perform. 22:3563–81
     [Google Scholar]
  56. Medendorp WP, Goltz HC, Vilis T, Crawford JD. 2003. Gaze-centered updating of visual space in human parietal cortex. J. Neurosci. 23:156209–14
     [Google Scholar]
  57. Melcher D. 2007. Predictive remapping of visual features precedes saccadic eye movements. Nat. Neurosci. 10:7903–7
     [Google Scholar]
  58. Melcher D, Morrone MC. 2003. Spatiotopic temporal integration of visual motion across saccadic eye movements. Nat. Neurosci. 6:8877–81
     [Google Scholar]
  59. Merriam EP, Genovese CR, Colby CL. 2003. Spatial updating in human parietal cortex. Neuron 39:2361–73
     [Google Scholar]
  60. Merriam EP, Genovese CR, Colby CL. 2007. Remapping in human visual cortex. J. Neurophysiol. 97:21738–55
     [Google Scholar]
  61. Mirpour K, Bisley JW. 2012. Anticipatory remapping of attentional priority across the entire visual field. J. Neurosci. 32:4616449–57
     [Google Scholar]
  62. Morris AP, Liu CC, Cropper SJ, Forte JD, Krekelberg B, Mattingley JB. 2010. Summation of visual motion across eye movements reflects a nonspatial decision mechanism. J. Neurosci. 30:299821–30
     [Google Scholar]
  63. Nakamura K, Colby CL 2002. Updating of the visual representation in monkey striate and extrastriate cortex during saccades. PNAS 99:64026–31
     [Google Scholar]
  64. Neupane S, Guitton D, Pack CC. 2016a. Dissociation of forward and convergent remapping in primate visual cortex. Curr. Biol. 26:12R491–92
     [Google Scholar]
  65. Neupane S, Guitton D, Pack CC. 2016b. Two distinct types of remapping in primate cortical area V4. Nat. Commun. 7:10402
     [Google Scholar]
  66. Neupane S, Guitton D, Pack CC. 2020. Perisaccadic remapping: What? How? Why?. Rev. Neurosci. 31:5505–20
     [Google Scholar]
  67. Olshausen BA, Anderson CH, Van Essen DC. 1993. A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information. J. Neurosci. 13:114700–19
     [Google Scholar]
  68. Ong WS, Bisley JW. 2011. A lack of anticipatory remapping of retinotopic receptive fields in the middle temporal area. J. Neurosci. 31:2910432–36
     [Google Scholar]
  69. Oostwoud Wijdenes L, Marshall L, Bays PM 2015. Evidence for optimal integration of visual feature representations across saccades. J. Neurosci. 35:2810146–53
     [Google Scholar]
  70. O'Regan JK, Lévy-Schoen A. 1983. Integrating visual information from successive fixations: Does trans-saccadic fusion exist?. Vis. Res. 23:8765–68
     [Google Scholar]
  71. Paeye C, Collins T, Cavanagh P. 2017. Transsaccadic perceptual fusion. J. Vis. 17:114
     [Google Scholar]
  72. Parks NA, Corballis PM. 2008. Electrophysiological correlates of presaccadic remapping in humans. Psychophysiology 45:5776–83
     [Google Scholar]
  73. Quaia C, Optican LM, Goldberg ME. 1998. The maintenance of spatial accuracy by the perisaccadic remapping of visual receptive fields. Neural Netw 11:7–81229–40
     [Google Scholar]
  74. Rao HM, Mayo JP, Sommer MA. 2016a. Circuits for presaccadic visual remapping. J. Neurophysiol. 116:62624–36
     [Google Scholar]
  75. Rao HM, San Juan J, Shen FY, Villa JE, Rafie KS, Sommer MA 2016b. Neural network evidence for the coupling of presaccadic visual remapping to predictive eye position updating. Front. Comput. Neurosci. 10:52
     [Google Scholar]
  76. Reynolds JH, Desimone R. 1999. The role of neural mechanisms of attention in solving the binding problem. Neuron 24:119–29
     [Google Scholar]
  77. Reynolds JH, Heeger DJ. 2009. The normalization model of attention. Neuron 61:2168–85
     [Google Scholar]
  78. Rizzolatti G, Riggio L, Dascola I, Umilta C. 1987. Reorienting attention across the horizontal and vertical meridians: evidence in favor of a premotor theory of attention. Neuropsychologia 25:1A31–40
     [Google Scholar]
  79. Rolfs M, Jonikaitis D, Deubel H, Cavanagh P. 2011. Predictive remapping of attention across eye movements. Nat. Neurosci. 14:2252–56
     [Google Scholar]
  80. Sereno AB, Maunsell JH. 1998. Shape selectivity in primate lateral intraparietal cortex. Nature 395:6701500–3
     [Google Scholar]
  81. Shafer-Skelton A, Golomb JD. 2017. Memory for retinotopic locations is more accurate than memory for spatiotopic locations, even for visually guided reaching. Psychon. Bull. Rev. 25:1388–98
     [Google Scholar]
  82. Shafer-Skelton A, Kupitz CN, Golomb JD. 2017. Object-location binding across a saccade: a retinotopic spatial congruency bias. Atten. Percept. Psychophys. 79:3765–81
     [Google Scholar]
  83. Sommer MA, Wurtz RH. 2002. A pathway in primate brain for internal monitoring of movements. Science 296:55721480–82
     [Google Scholar]
  84. Sommer MA, Wurtz RH. 2006. Influence of the thalamus on spatial visual processing in frontal cortex. Nature 444:7117374–77
     [Google Scholar]
  85. Sommer MA, Wurtz RH. 2008. Brain circuits for the internal monitoring of movements. Annu. Rev. Neurosci. 31:317–38
     [Google Scholar]
  86. Subramanian J, Colby CL. 2013. Shape selectivity and remapping in dorsal stream visual area LIP. J. Neurophysiol. 111:3613–27
     [Google Scholar]
  87. Sun LD, Goldberg ME. 2016. Corollary discharge and oculomotor proprioception: cortical mechanisms for spatially accurate vision. Annu. Rev. Vis. Sci. 2:61–84
     [Google Scholar]
  88. Szinte M, Jonikaitis D, Rangelov D, Deubel H 2018. Pre-saccadic remapping relies on dynamics of spatial attention. eLife 7:e37598
     [Google Scholar]
  89. Talsma D, White BJ, Mathôt S, Munoz DP, Theeuwes J. 2013. A retinotopic attentional trace after saccadic eye movements: evidence from event-related potentials. J. Cogn. Neurosci. 25:91563–77
     [Google Scholar]
  90. Tolias AS, Moor T, Smirnikas SM, Tehovnik EJ, Siapas AG, Schiller PH. 2001. Eye movements modulate visual receptive fields of V4 neurons. Neuron 29:757–67
     [Google Scholar]
  91. Treisman AM. 1996. The binding problem. Curr. Opin. Neurobiol. 6:2171–78
     [Google Scholar]
  92. Umeno MM, Goldberg ME. 1997. Spatial processing in the monkey frontal eye field. I. Predictive visual responses. J. Neurophysiol. 78:31373–83
     [Google Scholar]
  93. Umeno MM, Goldberg ME. 2001. Spatial processing in the monkey frontal eye field. II. Memory responses. J. Neurophysiol. 86:52344–52
     [Google Scholar]
  94. Ungerleider LG, Mishkin M 1982. Two cortical visual systems. Analysis of Visual Behavior DJ Ingle, MA Goodale, RJW Mansfield 529–86 Cambridge, MA: MIT Press
     [Google Scholar]
  95. Vinje WE, Gallant JL. 2000. Sparse coding and decorrelation in primary visual cortex during natural vision. Science 287:1273–76
     [Google Scholar]
  96. von der Malsburg C. 1999. The what and why of binding: the modeler's perspective. Neuron 24:195–104
     [Google Scholar]
  97. Walker MF, Fitzgibbon EJ, Goldberg ME. 1995. Neurons in the monkey superior colliculus predict the visual result of impending saccadic eye movements. J. Neurophys. 73:51988–2003
     [Google Scholar]
  98. Wenderoth P, Wiese M. 2008. Retinotopic encoding of the direction aftereffect. Vis. Res. 48:191949–54
     [Google Scholar]
  99. Williams MA, Baker CI, Op de Beeck HP, Mok Shim W, Dang S et al. 2008. Feedback of visual object information to foveal retinotopic cortex. Nat. Neurosci. 11:121439–45
     [Google Scholar]
  100. Wolfe BA, Whitney D. 2015. Saccadic remapping of object-selective information. Atten. Percept. Psychophys. 77:72260–69
     [Google Scholar]
  101. Wolfe JM, Cave KR. 1999. The psychophysical evidence for a binding problem in human vision. Neuron 24:1 11–17 111–25
     [Google Scholar]
  102. Womelsdorf T, Anton-Erxleben K, Pieper F, Treue S. 2006. Dynamic shifts of visual receptive fields in cortical area MT by spatial attention. Nat. Neurosci. 9:91156–60
     [Google Scholar]
  103. Wurtz RH. 2008. Neuronal mechanisms of visual stability. Vis. Res. 48:202070–89
     [Google Scholar]
  104. Yao T, Ketkar M, Treue S, Krishna BS 2016a. Visual attention is available at a task-relevant location rapidly after a saccade. eLife 5:e18009
     [Google Scholar]
  105. Yao T, Treue S, Krishna BS. 2016b. An attention-sensitive memory trace in macaque MT following saccadic eye movements. PLOS Biol 14:2e1002390
     [Google Scholar]
  106. Yao T, Treue S, Krishna BS. 2018. Saccade-synchronized rapid attention shifts in macaque visual cortical area MT. Nat. Commun. 9:1958
     [Google Scholar]
  107. Zhu SD, Zhang LA, von der Heydt R. 2020. Searching for object pointers in the visual cortex. J. Neurophysiol. 123:51979–94
     [Google Scholar]
  108. Ziesche A, Bergelt J, Deubel H, Hamker FH. 2017. Pre- and post-saccadic stimulus timing in saccadic suppression of displacement: a computational model. Vis. Res. 138:1–11
     [Google Scholar]
  109. Ziesche A, Hamker FH. 2011. A computational model for the influence of corollary discharge and proprioception on the perisaccadic mislocalization of briefly presented stimuli in complete darkness. J. Neurosci. 31:4817392–405
     [Google Scholar]
  110. Zimmermann E, Morrone MC, Fink GR, Burr D. 2013. Spatiotopic neural representations develop slowly across saccades. Curr. Biol. 23:5R193–94
     [Google Scholar]
  111. Zimmermann E, Weidner R, Abdollahi RO, Fink GR. 2016. Spatiotopic adaptation in visual areas. J. Neurosci. 36:379526–34
     [Google Scholar]
  112. Zirnsak M, Gerhards RGK, Kiani R, Lappe M, Hamker FH. 2011. Anticipatory saccade target processing and the presaccadic transfer of visual features. J. Neurosci. 31:4917887–91
     [Google Scholar]
  113. Zirnsak M, Moore T. 2014. Saccades and shifting receptive fields: anticipating consequences or selecting targets?. Trends Cogn. Sci. 18:12621–28
     [Google Scholar]
  114. Zirnsak M, Steinmetz NA, Noudoost B, Xu KZ, Moore T 2014. Visual space is compressed in prefrontal cortex before eye movements. Nature 507:7493504–7
     [Google Scholar]
/content/journals/10.1146/annurev-vision-032321-100012
Loading
Visual Remapping
Annual Review of Vision Science 7, 257 (2021); https://doi.org/10.1146/annurev-vision-032321-100012
/content/journals/10.1146/annurev-vision-032321-100012
/content/journals/10.1146/annurev-vision-032321-100012
Loading

Data & Media loading...

  • Article Type: Review Article

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error