1932

Abstract

Adaptive behavior in a complex, dynamic, and multisensory world poses some of the most fundamental computational challenges for the brain, notably inference, decision-making, learning, binding, and attention. We first discuss how the brain integrates sensory signals from the same source to support perceptual inference and decision-making by weighting them according to their momentary sensory uncertainties. We then show how observers solve the binding or causal inference problem—deciding whether signals come from common causes and should hence be integrated or else be treated independently. Next, we describe the multifarious interplay between multisensory processing and attention. We argue that attentional mechanisms are crucial to compute approximate solutions to the binding problem in naturalistic environments when complex time-varying signals arise from myriad causes. Finally, we review how the brain dynamically adapts multisensory processing to a changing world across multiple timescales.

Loading

Article metrics loading...

/content/journals/10.1146/annurev-neuro-100120-085519
2021-07-08
2024-06-19
Loading full text...

Full text loading...

/deliver/fulltext/neuro/44/1/annurev-neuro-100120-085519.html?itemId=/content/journals/10.1146/annurev-neuro-100120-085519&mimeType=html&fmt=ahah

Literature Cited

  1. Acerbi L, Dokka K, Angelaki DE, Ma WJ. 2018. Bayesian comparison of explicit and implicit causal inference strategies in multisensory heading perception. PLOS Comput. Biol. 14:e1006110
    [Google Scholar]
  2. Alais D, Burr D. 2004. Ventriloquist effect results from near-optimal bimodal integration. Curr. Biol. 14:257–62
    [Google Scholar]
  3. Aller M, Mihalik A, Noppeney U 2021. Audiovisual adaptation is expressed in spatial and decisional codes. bioRxiv 2021.02.15.431309 https://doi.org/10.1101/2021.02.15.431309
    [Crossref] [Google Scholar]
  4. Aller M, Noppeney U. 2019. To integrate or not to integrate: temporal dynamics of hierarchical Bayesian causal inference. PLOS Biol 17:e3000210
    [Google Scholar]
  5. Alsius A, Möttönen R, Sams ME, Soto-Faraco S, Tiippana K. 2014. Effect of attentional load on audiovisual speech perception: evidence from ERPs. Front. Psychol. 5:727
    [Google Scholar]
  6. Alsius A, Navarra J, Campbell R, Soto-Faraco S. 2005. Audiovisual integration of speech falters under high attention demands. Curr. Biol. 15:839–43
    [Google Scholar]
  7. Alsius A, Soto-Faraco S. 2011. Searching for audiovisual correspondence in multiple speaker scenarios. Exp. Brain Res. 213:175–83
    [Google Scholar]
  8. Atilgan H, Town SM, Wood KC, Jones GP, Maddox RK et al. 2018. Integration of visual information in auditory cortex promotes auditory scene analysis through multisensory binding. Neuron 97:640–55.e4
    [Google Scholar]
  9. Badde S, Navarro KT, Landy MS. 2020. Modality-specific attention attenuates visual-tactile integration and recalibration effects by reducing prior expectations of a common source for vision and touch. Cognition 197:104170
    [Google Scholar]
  10. Bankieris KR, Bejjanki VR, Aslin RN. 2017. Sensory cue-combination in the context of newly learned categories. Sci. Rep. 7:10890
    [Google Scholar]
  11. Bao M, Engel SA 2012. Distinct mechanism for long-term contrast adaptation. PNAS 109:5898–903
    [Google Scholar]
  12. Battaglia PW, Jacobs RA, Aslin RN. 2003. Bayesian integration of visual and auditory signals for spatial localization. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 20:1391–97
    [Google Scholar]
  13. Beck JM, Ma WJ, Kiani R, Hanks T, Churchland AK et al. 2008. Probabilistic population codes for Bayesian decision making. Neuron 60:1142–52
    [Google Scholar]
  14. Beierholm U, Rohe T, Ferrari A, Stegle O, Noppeney U. 2020. Using the past to estimate sensory uncertainty. eLife 9:e54172
    [Google Scholar]
  15. Bejjanki VR, Clayards M, Knill DC, Aslin RN. 2011. Cue integration in categorical tasks: insights from audio-visual speech perception. PLOS ONE 6:e19812
    [Google Scholar]
  16. Bertelson P, Frissen I, Vroomen J, de Gelder B. 2006. The aftereffects of ventriloquism: patterns of spatial generalization. Percept. Psychophys. 68:428–36
    [Google Scholar]
  17. Bertelson P, Radeau M. 1981. Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Percept. Psychophys. 29:578–84
    [Google Scholar]
  18. Bertelson P, Vroomen J, de Gelder B. 2003. Visual recalibration of auditory speech identification: a McGurk aftereffect. Psychol. Sci. 14:592–97
    [Google Scholar]
  19. Bertelson P, Vroomen J, de Gelder B, Driver J. 2000. The ventriloquist effect does not depend on the direction of deliberate visual attention. Percept. Psychophys. 62:321–32
    [Google Scholar]
  20. Bertelson P, Vroomen J, Wiegeraad G, de Gelder B. 1994. Exploring the relation between McGurk interference and ventriloquism. Proceedings of the Third International Congress on Spoken Language Processing (ICSLP 94) Yokohama, Japan: September 18–22, 1994 559–62 Baixas, Fr.: Int. Speech Commun. Assoc .
    [Google Scholar]
  21. Bishop CW, Miller LM. 2011. Speech cues contribute to audiovisual spatial integration. PLOS ONE 6:e24016
    [Google Scholar]
  22. Bizley JK, Nodal FR, Bajo VM, Nelken I, King AJ. 2007. Physiological and anatomical evidence for multisensory interactions in auditory cortex. Cereb. Cortex 17:2172–89
    [Google Scholar]
  23. Bizley JK, Shinn-Cunningham BG, Lee AK 2012. Nothing is irrelevant in a noisy world: Sensory illusions reveal obligatory within- and across-modality integration. J. Neurosci. 32:13402–10
    [Google Scholar]
  24. Bonath B, Noesselt T, Martinez A, Mishra J, Schwiecker K et al. 2007. Neural basis of the ventriloquist illusion. Curr. Biol. 17:1697–703
    [Google Scholar]
  25. Bosen AK, Fleming JT, Allen PD, O'Neill WE, Paige GD. 2017. Accumulation and decay of visual capture and the ventriloquism aftereffect caused by brief audio-visual disparities. Exp. Brain Res. 235:585–95
    [Google Scholar]
  26. Bosen AK, Fleming JT, Allen PD, O'Neill WE, Paige GD 2018. Multiple time scales of the ventriloquism aftereffect. PLOS ONE 13:e0200930
    [Google Scholar]
  27. Bruns P, Röder B. 2015. Sensory recalibration integrates information from the immediate and the cumulative past. Sci. Rep. 5:12739
    [Google Scholar]
  28. Burge J, Girshick AR, Banks MS. 2010. Visual-haptic adaptation is determined by relative reliability. J. Neurosci. 30:7714–21
    [Google Scholar]
  29. Burr D, Banks MS, Morrone MC. 2009. Auditory dominance over vision in the perception of interval duration. Exp. Brain Res. 198:49–57
    [Google Scholar]
  30. Busse L, Roberts KC, Crist RE, Weissman DH Woldorff MG. 2005. The spread of attention across modalities and space in a multisensory object italicPNAS 102:18751–56
    [Google Scholar]
  31. Butler JS, Smith ST, Campos JL, Bülthoff HH. 2010. Bayesian integration of visual and vestibular signals for heading. J. Vis. 10:23
    [Google Scholar]
  32. Calvert GA, Campbell R, Brammer MJ. 2000. Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr. Biol. 10:11649–57
    [Google Scholar]
  33. Cao Y, Summerfield C, Park H, Giordano BL, Kayser C. 2019. Causal inference in the multisensory brain. Neuron 102:1076–87.e8
    [Google Scholar]
  34. Carandini M, Heeger DJ. 2011. Normalization as a canonical neural computation. Nat. Rev. Neurosci. 13:51–62
    [Google Scholar]
  35. Chen L, Vroomen J. 2013. Intersensory binding across space and time: a tutorial review. Atten. Percept. Psychophys. 75:790–811
    [Google Scholar]
  36. Chen Y-C, Spence C. 2017. Assessing the role of the ‘unity assumption’ on multisensory integration: a review. Front. Psychol. 8:445
    [Google Scholar]
  37. Crosse MJ, Butler JS, Lalor EC. 2015. Congruent visual speech enhances cortical entrainment to continuous auditory speech in noise-free conditions. J. Neurosci. 35:14195–204
    [Google Scholar]
  38. Dayan P, Kakade S, Montague PR. 2000. Learning and selective attention. Nat. Neurosci. 3:1218–23
    [Google Scholar]
  39. Dayan P, Solomon JA. 2010. Selective Bayes: attentional load and crowding. Vis. Res. 50:2248–60
    [Google Scholar]
  40. de Winkel KN, Katliar M, Diers D, Bulthoff HH. 2018. Causal inference in the perception of verticality. Sci. Rep. 8:15483
    [Google Scholar]
  41. Delong P, Noppeney U 2021. Semantic and spatial congruency mould audiovisual integration depending on perceptual awareness. Sci. Rep In press
    [Google Scholar]
  42. Di Luca M, Machulla TK, Ernst MO. 2009. Recalibration of multisensory simultaneity: Cross-modal transfer coincides with a change in perceptual latency. J. Vis. 9:7
    [Google Scholar]
  43. Donohue SE, Roberts KC, Grent-’t-Jong T, Woldorff MG 2011. The cross-modal spread of attention reveals differential constraints for the temporal and spatial linking of visual and auditory stimulus events. J. Neurosci. 31:7982–90
    [Google Scholar]
  44. Drugowitsch J, DeAngelis GC, Angelaki DE, Pouget A. 2015. Tuning the speed-accuracy trade-off to maximize reward rate in multisensory decision-making. eLife 4:e06678
    [Google Scholar]
  45. Drugowitsch J, DeAngelis GC, Klier EM, Angelaki DE, Pouget A. 2014. Optimal multisensory decision-making in a reaction-time task. eLife 3:e03005
    [Google Scholar]
  46. Eimer M, van Velzen J, Driver J. 2004. ERP evidence for cross-modal audiovisual effects of endogenous spatial attention within hemifields. J. Cogn. Neurosci. 16:272–88
    [Google Scholar]
  47. Ernst MO. 2007. Learning to integrate arbitrary signals from vision and touch. J. Vis. 7:7
    [Google Scholar]
  48. Ernst MO, Banks MS. 2002. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415:429–33
    [Google Scholar]
  49. Ernst MO, Bülthoff HH. 2004. Merging the senses into a robust percept. Trends Cogn. Sci. 8:162–69
    [Google Scholar]
  50. Ernst MO, di Luca M 2011. Multisensory perception: from integration to remapping. Sensory Cue Integration ed. J Trommershäuser, KP Kording, MS Landy 225–50 Oxford, UK: Oxford Univ. Press
    [Google Scholar]
  51. Fairhall SL, Macaluso E. 2009. Spatial attention can modulate audiovisual integration at multiple cortical and subcortical sites. Eur. J. Neurosci. 29:1247–57
    [Google Scholar]
  52. Faisal AA, Selen LP, Wolpert DM. 2008. Noise in the nervous system. Nat. Rev. Neurosci. 9:292–303
    [Google Scholar]
  53. Fetsch CR, DeAngelis GC, Angelaki DE. 2013. Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons. Nat. Rev. Neurosci. 14:429–42
    [Google Scholar]
  54. Fetsch CR, Pouget A, DeAngelis GC, Angelaki DE. 2012. Neural correlates of reliability-based cue weighting during multisensory integration. Nat. Neurosci. 15:146–54
    [Google Scholar]
  55. Fetsch CR, Turner AH, DeAngelis GC, Angelaki DE. 2009. Dynamic reweighting of visual and vestibular cues during self-motion perception. J. Neurosci. 29:15601–12
    [Google Scholar]
  56. Fiebelkorn IC, Foxe JJ, Molholm S. 2010. Dual mechanisms for the cross-sensory spread of attention: How much do learned associations matter?. Cereb. Cortex 20:109–20
    [Google Scholar]
  57. Forstmann BU, Ratcliff R, Wagenmakers EJ. 2016. Sequential sampling models in cognitive neuroscience: advantages, applications, and extensions. Annu. Rev. Psychol. 67:641–66
    [Google Scholar]
  58. Fujisaki W, Koene A, Arnold D, Johnston A, Nishida S. 2006. Visual search for a target changing in synchrony with an auditory signal. Proc. Biol. Sci. 273:865–74
    [Google Scholar]
  59. Fujisaki W, Shimojo S, Kashino M, Nishida S. 2004. Recalibration of audiovisual simultaneity. Nat. Neurosci. 7:773–78
    [Google Scholar]
  60. Gau R, Bazin PL, Trampel R, Turner R, Noppeney U. 2020. Resolving multisensory and attentional influences across cortical depth in sensory cortices. eLife 9:e46856
    [Google Scholar]
  61. Gau R, Noppeney U. 2016. How prior expectations shape multisensory perception. NeuroImage 124:876–86
    [Google Scholar]
  62. Gepshtein S, Banks MS. 2003. Viewing geometry determines how vision and haptics combine in size perception. Curr. Biol. 13:483–88
    [Google Scholar]
  63. Gharamani Z, Wolpert DM, Jordan MI 1997. Computational models of sensorimotor integration. Advances in Psychology, Vol. 119: Self-Organization, Computational Maps, and Motor Control ed. P. Morasso, V Sanguineti 117–47 Amsterdam: Elsevier
    [Google Scholar]
  64. Ghazanfar A, Schroeder C. 2006. Is neocortex essentially multisensory?. Trends Cogn. Sci. 10:278–85
    [Google Scholar]
  65. Gold JI, Shadlen MN. 2007. The neural basis of decision making. Annu. Rev. Neurosci. 30:535–74
    [Google Scholar]
  66. Gu Y, Angelaki DE, DeAngelis GC. 2008. Neural correlates of multisensory cue integration in macaque MSTd. Nat. Neurosci. 11:1201–10
    [Google Scholar]
  67. Gu Y, DeAngelis GC, Angelaki DE. 2012. Causal links between dorsal medial superior temporal area neurons and multisensory heading perception. J. Neurosci. 32:2299–313
    [Google Scholar]
  68. Hein G, Doehrmann O, Müller NG, Kaiser J, Muckli L, Naumer MJ. 2007. Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. J. Neurosci. 27:7881–87
    [Google Scholar]
  69. Helbig HB, Ernst MO, Ricciardi E, Pietrini P, Thielscher A et al. 2012. The neural mechanisms of reliability weighted integration of shape information from vision and touch. NeuroImage 60:1063–72
    [Google Scholar]
  70. Hillis JM, Ernst MO, Banks MS, Landy MS. 2002. Combining sensory information: mandatory fusion within, but not between, senses. Science 298:1627–30
    [Google Scholar]
  71. Hospedales T, Vijayakumar S. 2009. Multisensory oddity detection as Bayesian inference. PLOS ONE 4:e4205
    [Google Scholar]
  72. Hou H, Zheng Q, Zhao Y, Pouget A, Gu Y. 2019. Neural correlates of optimal multisensory decision making under time-varying reliabilities with an invariant linear probabilistic population Code. Neuron 104:1010–21.e10
    [Google Scholar]
  73. Itti L, Koch C. 2001. Computational modelling of visual attention. Nat. Rev. Neurosci. 2:194–203
    [Google Scholar]
  74. Jones SA, Beierholm U, Meijer D, Noppeney U. 2019. Older adults sacrifice response speed to preserve multisensory integration performance. Neurobiol. Aging 84:148–57
    [Google Scholar]
  75. Kaliuzhna M, Prsa M, Gale S, Lee SJ, Blanke O. 2015. Learning to integrate contradictory multisensory self-motion cue pairings. J. Vis. 15:10
    [Google Scholar]
  76. Kanaya S, Yokosawa K. 2011. Perceptual congruency of audio-visual speech affects ventriloquism with bilateral visual stimuli. Psychon. Bull. Rev. 18:123–28
    [Google Scholar]
  77. Kayser C, Petkov CI, Augath M, Logothetis NK. 2007. Functional imaging reveals visual modulation of specific fields in auditory cortex. J. Neurosci. 27:1824–35
    [Google Scholar]
  78. Kayser C, Petkov CI, Lippert M, Logothetis NK. 2005. Mechanisms for allocating auditory attention: an auditory saliency map. Curr. Biol. 15:1943–47
    [Google Scholar]
  79. Kayser C, Petkov CI, Logothetis NK. 2008. Visual modulation of neurons in auditory cortex. Cereb. Cortex 18:1560–74
    [Google Scholar]
  80. Kerns JG, Cohen JD, MacDonald AW 3rd, Cho RY, Stenger VA, Carter CS. 2004. Anterior cingulate conflict monitoring and adjustments in control. Science 303:1023–26
    [Google Scholar]
  81. Kersten D, Mamassian P, Yuille A. 2004. Object perception as Bayesian inference. Annu. Rev. Psychol. 55:271–304
    [Google Scholar]
  82. Kilian-Hütten N, Valente G, Vroomen J, Formisano E. 2011. Auditory cortex encodes the perceptual interpretation of ambiguous sound. J. Neurosci. 31:1715–20
    [Google Scholar]
  83. Knill DC, Saunders JA. 2003. Do humans optimally integrate stereo and texture information for judgments of surface slant?. Vis. Res. 43:2539–58
    [Google Scholar]
  84. Kopčo N, Groh JM, Lin IF, Shinn-Cunningham BG. 2009. Reference frame of the ventriloquism aftereffect. J. Neurosci. 29:13809–14
    [Google Scholar]
  85. Körding KP, Beierholm U, Ma WJ, Quartz S, Tenenbaum JB, Shams L. 2007a. Causal inference in multisensory perception. PLOS ONE 2:e943
    [Google Scholar]
  86. Kording KP, Tenenbaum JB, Shadmehr R. 2007b. The dynamics of memory as a consequence of optimal adaptation to a changing body. Nat. Neurosci. 10:779–86
    [Google Scholar]
  87. Kumpik DP, Roberts HE, King AJ, Bizley JK. 2014. Visual sensitivity is a stronger determinant of illusory processes than auditory cue parameters in the sound-induced flash illusion. J. Vis. 14:12
    [Google Scholar]
  88. Lakatos P, Chen C-M, O'Connell MN, Mills A, Schroeder CE 2007. Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53:279–92
    [Google Scholar]
  89. Lee H, Noppeney U. 2011a. Long-term music training tunes how the brain temporally binds signals from multiple senses italicPNAS 108:E1441–50
    [Google Scholar]
  90. Lee H, Noppeney U. 2011b. Physical and perceptual factors shape the neural mechanisms that integrate audiovisual signals in speech comprehension. J. Neurosci. 31:11338–50
    [Google Scholar]
  91. Lee H, Noppeney U. 2014. Temporal prediction errors in visual and auditory cortices. Curr. Biol 24:R30910
    [Google Scholar]
  92. Lewald J, Guski R. 2003. Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli. Brain Res. Cogn. Brain Res. 16:468–78
    [Google Scholar]
  93. Lewis R, Noppeney U. 2010. Audiovisual synchrony improves motion discrimination via enhanced connectivity between early visual and auditory areas. J. Neurosci. 30:12329–39
    [Google Scholar]
  94. Li Z. 2002. A saliency map in primary visual cortex. Trends Cogn. Sci. 6:9–16
    [Google Scholar]
  95. Locke SM, Landy MS. 2017. Temporal causal inference with stochastic audiovisual sequences. PLOS ONE 12:e0183776
    [Google Scholar]
  96. Ma WJ. 2012. Organizing probabilistic models of perception. Trends Cogn. Sci. 16:511–18
    [Google Scholar]
  97. Ma WJ, Beck JM, Latham PE, Pouget A. 2006. Bayesian inference with probabilistic population codes. Nat. Neurosci. 9:1432–38
    [Google Scholar]
  98. Ma WJ, Jazayeri M. 2014. Neural coding of uncertainty and probability. Annu. Rev. Neurosci. 37:205–20
    [Google Scholar]
  99. Magnotti JF, Beauchamp MS. 2017. A causal inference model explains perception of the McGurk effect and other incongruent audiovisual speech. PLOS Comput. Biol. 13:e1005229
    [Google Scholar]
  100. Magnotti JF, Ma WJ, Beauchamp MS. 2013. Causal inference of asynchronous audiovisual speech. Front. Psychol. 4:798
    [Google Scholar]
  101. Maier JX, Groh JM. 2009. Multisensory guidance of orienting behavior. Hear. Res. 258:106–12
    [Google Scholar]
  102. Martuzzi R, Murray MM, Michel CM, Thiran JP, Maeder PP et al. 2007. Multisensory interactions within human primary cortices revealed by BOLD dynamics. Cereb. Cortex 17:1672–79
    [Google Scholar]
  103. Maunsell JHR. 2015. Neuronal mechanisms of visual attention. Annu. Rev. Vis. Sci. 1:373–91
    [Google Scholar]
  104. McGurk H, MacDonald J. 1976. Hearing lips and seeing voices. Nature 264:691–811
    [Google Scholar]
  105. Meijer D, Noppeney U 2020. Computational models of multisensory integration. Multisensory Perception: From Laboratory to Clinic ed. K Sathian, VS Ramachandran 113–33 London: Academic Press
    [Google Scholar]
  106. Meijer D, Veselič S, Calafiore C, Noppeney U. 2019. Integration of audiovisual spatial signals is not consistent with maximum likelihood estimation. Cortex 119:74–88
    [Google Scholar]
  107. Mengotti P, Boers F, Dombert PL, Fink GR, Vossel S. 2018. Integrating modality-specific expectancies for the deployment of spatial attention. Sci. Rep. 8:1210
    [Google Scholar]
  108. Mercier MR, Foxe JJ, Fiebelkorn IC, Butler JS, Schwartz TH, Molholm S. 2013. Auditory-driven phase reset in visual cortex: Human electrocorticography reveals mechanisms of early multisensory integration. Neuroimage 79:19–29
    [Google Scholar]
  109. Meredith MA, Nemitz JW, Stein BE. 1987. Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. J. Neurosci. 7:3215–29
    [Google Scholar]
  110. Meredith MA, Stein B. 1983. Interactions among converging sensory inputs in the superior colliculus. Science 221:389–91
    [Google Scholar]
  111. Metzger BA, Magnotti JF, Wang Z, Nesbitt E, Karas PJ et al. 2020. Responses to visual speech in human posterior superior temporal gyrus examined with iEEG deconvolution. J. Neurosci. 40:6938–48
    [Google Scholar]
  112. Mihalik A, Noppeney U. 2020. Causal inference in audiovisual perception. J. Neurosci. 40:6600–12
    [Google Scholar]
  113. Miller J. 1982. Divided attention: evidence for coactivation with redundant signals. Cogn. Psychol. 14:247–79
    [Google Scholar]
  114. Miller LM, D'Esposito M 2005. Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J. Neurosci. 25:5884–93
    [Google Scholar]
  115. Mohl JT, Pearson JM, Groh JM. 2020. Monkeys and humans implement causal inference to simultaneously localize auditory and visual stimuli. J. Neurophysiol. 124:715–27
    [Google Scholar]
  116. Nahorna O, Berthommier F, Schwartz J-L. 2012. Binding and unbinding the auditory and visual streams in the McGurk effect. J. Acoust. Soc. Am. 132:1061–77
    [Google Scholar]
  117. Nahorna O, Berthommier F, Schwartz J-L. 2015. Audio-visual speech scene analysis: characterization of the dynamics of unbinding and rebinding the McGurk effect. J. Acoust. Soc. Am. 137:362–77
    [Google Scholar]
  118. Nath AR, Beauchamp MS. 2011. Dynamic changes in superior temporal sulcus connectivity during perception of noisy audiovisual speech. J. Neurosci. 31:1704–14
    [Google Scholar]
  119. Negen J, Wen L, Thaler L, Nardini M. 2018. Bayes-like integration of a new sensory skill with vision. Sci. Rep. 8:16880
    [Google Scholar]
  120. Nikbakht N, Tafreshiha A, Zoccolan D, Diamond ME. 2018. Supralinear and supramodal integration of visual and tactile signals in rats: psychophysics and neuronal mechanisms. Neuron 97:626–39.e8
    [Google Scholar]
  121. Noesselt T, Rieger JW, Schoenfeld MA, Kanowski M, Hinrichs H et al. 2007. Audiovisual temporal correspondence modulates human multisensory superior temporal sulcus plus primary sensory cortices. J. Neurosci. 27:11431–41
    [Google Scholar]
  122. Noppeney U 2020. Multisensory perception: behaviour, computations, neural mechanisms. The Cognitive Neurosciences ed. D Poeppel, GR Mangun, M Gazzaniga 141–51 Cambridge, MA: MIT Press
    [Google Scholar]
  123. Noppeney U, Josephs O, Hocking J, Price CJ, Friston KJ. 2008. The effect of prior visual information on recognition of speech and sounds. Cereb. Cortex 18:598–609
    [Google Scholar]
  124. Noppeney U, Ostwald D, Werner S 2010. Perceptual decisions formed by accumulation of audiovisual evidence in prefrontal cortex. J. Neurosci. 30:7434–46
    [Google Scholar]
  125. Ohshiro T, Angelaki DE, DeAngelis GC. 2011. A normalization model of multisensory integration. Nat. Neurosci. 14:775–82
    [Google Scholar]
  126. Ohshiro T, Angelaki DE, DeAngelis GC. 2017. A neural signature of divisive normalization at the level of multisensory integration in primate cortex. Neuron 95:399–411.e8
    [Google Scholar]
  127. Olasagasti I, Giraud AL. 2020. Integrating prediction errors at two time scales permits rapid recalibration of speech sound categories. eLife 9:e44516
    [Google Scholar]
  128. Ortiz-Rios M, Azevedo FAC, Kuśmierek P, Balla DZ, Munk MH et al. 2017. Widespread and opponent fMRI signals represent sound location in macaque auditory cortex. Neuron 93:971–83.e4
    [Google Scholar]
  129. O'Sullivan JA, Power AJ, Mesgarani N, Rajaram S, Foxe JJ et al. 2015. Attentional selection in a cocktail party environment can be decoded from single-trial EEG. Cereb. Cortex 25:1697–706
    [Google Scholar]
  130. Otto TU, Mamassian P. 2012. Noise and correlations in parallel perceptual decision making. Curr. Biol. 22:1391–96
    [Google Scholar]
  131. Parise CV, Ernst MO. 2016. Correlation detection as a general mechanism for multisensory integration. Nat. Commun. 7:11543
    [Google Scholar]
  132. Parise CV, Spence C, Ernst MO. 2012. When correlation implies causation in multisensory integration. Curr. Biol. 22:46–49
    [Google Scholar]
  133. Park H, Ince RAA, Schyns PG, Thut G, Gross J. 2018. Representational interactions during audiovisual speech entrainment: Redundancy in left posterior superior temporal gyrus and synergy in left motor cortex. PLOS Biol 16:e2006558
    [Google Scholar]
  134. Park H, Kayser C. 2019. Shared neural underpinnings of multisensory integration and trial-by-trial perceptual recalibration in humans. eLife 8:e47001
    [Google Scholar]
  135. Pouget A, Beck JM, Ma WJ, Latham PE. 2013. Probabilistic brains: knowns and unknowns. Nat. Neurosci. 16:1170–78
    [Google Scholar]
  136. Powers AR 3rd, Hevey MA, Wallace MT 2012. Neural correlates of multisensory perceptual learning. J. Neurosci. 32:6263–74
    [Google Scholar]
  137. Rahnev D, Denison RN. 2018. Suboptimality in perceptual decision making. Behav. Brain Sci. 41:e223
    [Google Scholar]
  138. Raposo D, Sheppard JP, Schrater PR, Churchland AK. 2012. Multisensory decision-making in rats and humans. J. Neurosci. 32:3726–35
    [Google Scholar]
  139. Rauschecker JP, Tian B. 2000. Mechanisms and streams for processing of “what” and “where” in auditory cortex. PNAS 97:11800–6
    [Google Scholar]
  140. Rohe T, Ehlis A-C, Noppeney U. 2019. The neural dynamics of hierarchical Bayesian causal inference in multisensory perception. Nat. Commun. 10:1907
    [Google Scholar]
  141. Rohe T, Noppeney U. 2015a. Cortical hierarchies perform Bayesian causal inference in multisensory perception. PLOS Biol 13:e1002073
    [Google Scholar]
  142. Rohe T, Noppeney U. 2015b. Sensory reliability shapes perceptual inference via two mechanisms. J. Vis. 15:22
    [Google Scholar]
  143. Rohe T, Noppeney U. 2016. Distinct computational principles govern multisensory integration in primary sensory and association cortices. Curr. Biol. 26:509–14
    [Google Scholar]
  144. Rohe T, Noppeney U. 2018. Reliability-weighted integration of audiovisual signals can be modulated by top-down attention. eNeuro 5: ENEURO.0315-17.2018
    [Google Scholar]
  145. Rosas P, Wagemans J, Ernst MO, Wichmann FA. 2005. Texture and haptic cues in slant discrimination: reliability-based cue weighting without statistically optimal cue combination. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 22:801–9
    [Google Scholar]
  146. Roseboom W, Arnold DH. 2011. Twice upon a time: multiple concurrent temporal recalibrations of audiovisual speech. Psychol. Sci. 22:872–77
    [Google Scholar]
  147. Sato Y, Toyoizumi T, Aihara K. 2007. Bayesian inference explains perception of unity and ventriloquism aftereffect: identification of common sources of audiovisual stimuli. Neural Comput 19:3335–55
    [Google Scholar]
  148. Schlack A, Sterbing-D'Angelo SJ, Hartung K, Hoffmann KP, Bremmer F 2005. Multisensory space representations in the macaque ventral intraparietal area. J. Neurosci. 25:4616–25
    [Google Scholar]
  149. Schroeder CE, Foxe JJ. 2002. The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. Brain Res. Cogn. Brain Res. 14:187–98
    [Google Scholar]
  150. Schwartz O, Hsu A, Dayan P. 2007. Space and time in visual context. Nat. Rev. Neurosci. 8:522–35
    [Google Scholar]
  151. Shams L, Beierholm UR. 2010. Causal inference in perception. Trends Cogn. Sci. 14:425–32
    [Google Scholar]
  152. Shams L, Ma WJ, Beierholm U. 2005. Sound-induced flash illusion as an optimal percept. Neuroreport 16:1923–27
    [Google Scholar]
  153. Shen S, Ma WJ. 2016. A detailed comparison of optimality and simplicity in perceptual decision making. Psychol. Rev. 123:452–80
    [Google Scholar]
  154. Simon DM, Nidiffer AR, Wallace MT. 2018. Single trial plasticity in evidence accumulation underlies rapid recalibration to asynchronous audiovisual speech. Sci. Rep. 8:12499
    [Google Scholar]
  155. Smith MA, Ghazizadeh A, Shadmehr R. 2006. Interacting adaptive processes with different timescales underlie short-term motor learning. PLOS Biol 4:e179
    [Google Scholar]
  156. Spence C 2014. Orienting attention: a crossmodal perspective. The Oxford Handbook of Attention ed. AC Nobre, S Kastner 446–73 Oxford, UK: Oxford Univ. Press
    [Google Scholar]
  157. Spence C, Driver J. 1996. Audiovisual links in endogenous covert spatial attention. J. Exp. Psychol. Hum. Percept. Perform. 22:1005–30
    [Google Scholar]
  158. Stacey PC, Murphy T, Sumner CJ, Kitterick PT, Roberts KL. 2014. Searching for a talking face: the effect of degrading the auditory signal. J. Exp. Psychol. Hum. Percept. Perform. 40:2106–11
    [Google Scholar]
  159. Stanford TR, Quessy S, Stein BE. 2005. Evaluating the operations underlying multisensory integration in the cat superior colliculus. J. Neurosci. 25:6499–508
    [Google Scholar]
  160. Stein BE, Stanford TR. 2008. Multisensory integration: current issues from the perspective of the single neuron. Nat. Rev. Neurosci. 9:255–66
    [Google Scholar]
  161. Stocker A, Simoncelli E 2005. Sensory adaptation within a Bayesian framework for perception. Advances in Neural Information Processing Systems 18 (NIPS 2005), ed. Y Weiss, B Schölkopf, J Platt 1291–98 San Diego, CA: NeurIPS
    [Google Scholar]
  162. Summerfield C, Egner T. 2009. Expectation (and attention) in visual cognition. Trends Cogn. Sci. 13:9403–9
    [Google Scholar]
  163. Talsma D, Doty TJ, Woldorff MG. 2006. Selective attention and audiovisual integration: Is attending to both modalities a prerequisite for early integration?. Cereb. Cortex 17:679–90
    [Google Scholar]
  164. Talsma D, Senkowski D, Soto-Faraco S, Woldorff MG. 2010. The multifaceted interplay between attention and multisensory integration. Trends Cogn. Sci. 14:400–10
    [Google Scholar]
  165. Tiippana K, Puharinen H, Möttönen R, Sams M. 2011. Sound location can influence audiovisual speech perception when spatial attention is manipulated. Seeing Perceiving 24:67–90
    [Google Scholar]
  166. Treisman AM, Gelade G. 1980. A feature-integration theory of attention. Cogn. Psychol. 12:97–136
    [Google Scholar]
  167. Van der Burg E, Alais D, Cass J 2015. Audiovisual temporal recalibration occurs independently at two different time scales. Sci. Rep. 5:14526
    [Google Scholar]
  168. Van der Burg E, Cass J, Olivers CNL, Theeuwes J, Alais D. 2010. Efficient visual search from synchronized auditory signals requires transient audiovisual events. PLOS ONE 5:e10664
    [Google Scholar]
  169. Van der Burg E, Olivers CNL, Bronkhorst AW, Theeuwes J. 2008. Pip and pop: nonspatial auditory signals improve spatial visual search. J. Exp. Psychol. Hum. Percept. Perform. 34:1053–65
    [Google Scholar]
  170. Van der Burg E, Talsma D, Olivers CNL, Hickey C, Theeuwes J. 2011. Early multisensory interactions affect the competition among multiple visual objects. NeuroImage 55:1208–18
    [Google Scholar]
  171. Van Wanrooij MM, Bremen P, Van Opstal AJ 2010. Acquired prior knowledge modulates audiovisual integration. Eur. J. Neurosci. 31:1763–71
    [Google Scholar]
  172. van Wassenhove V, Grant KW, Poeppel D. 2007. Temporal window of integration in auditory-visual speech perception. Neuropsychologia 45:598–607
    [Google Scholar]
  173. von Helmholtz H. 1867. Handbuch der Physiologischen Optik Leipzig: Leopold Voss:
    [Google Scholar]
  174. Wallace MT, Roberson GE, Hairston WD, Stein BE, Vaughan JW, Schirillo JA. 2004. Unifying multisensory signals across time and space. Exp. Brain Res. 158:252–58
    [Google Scholar]
  175. Watson DM, Akeroyd MA, Roach NW, Webb BS. 2019. Distinct mechanisms govern recalibration to audio-visual discrepancies in remote and recent history. Sci. Rep. 9:8513
    [Google Scholar]
  176. Werner S, Noppeney U. 2010a. Distinct functional contributions of primary sensory and association areas to audiovisual integration in object categorization. J. Neurosci. 30:2662–75
    [Google Scholar]
  177. Werner S, Noppeney U. 2010b. Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization. Cereb. Cortex 20:1829–42
    [Google Scholar]
  178. Woods TM, Recanzone GH. 2004. Visually induced plasticity of auditory spatial perception in macaques. Curr. Biol. 14:1559–64
    [Google Scholar]
  179. Wozny DR, Beierholm UR, Shams L. 2010. Probability matching as a computational strategy used in perception. PLOS Comput. Biol. 6:e1000871
    [Google Scholar]
  180. Wozny DR, Shams L. 2011a. Computational characterization of visually induced auditory spatial adaptation. Front. Integr. Neurosci. 5:75
    [Google Scholar]
  181. Wozny DR, Shams L. 2011b. Recalibration of auditory space following milliseconds of cross-modal discrepancy. J. Neurosci. 31:4607–12
    [Google Scholar]
  182. Yu AJ, Dayan P, Cohen JD. 2009. Dynamics of attentional selection under conflict: toward a rational Bayesian account. J. Exp. Psychol. Hum. Percept. Perform. 35:700–17
    [Google Scholar]
  183. Zaidel A, Ma WJ, Angelaki DE. 2013. Supervised calibration relies on the multisensory percept. Neuron 80:1544–57
    [Google Scholar]
  184. Zaidel A, Turner AH, Angelaki DE. 2011. Multisensory calibration is independent of cue reliability. J. Neurosci. 31:13949–62
    [Google Scholar]
  185. Zierul B, Röder B, Tempelmann C, Bruns P, Noesselt T. 2017. The role of auditory cortex in the spatial ventriloquism aftereffect. Neuroimage 162:257–68
    [Google Scholar]
  186. Zion Golumbic EM, Ding N, Bickel S, Lakatos P, Schevon CA et al. 2013. Mechanisms underlying selective neuronal tracking of attended speech at a “cocktail party. .” Neuron 77:980–91
    [Google Scholar]
  187. Zuanazzi A, Noppeney U. 2019. Distinct neural mechanisms of spatial attention and expectation guide perceptual inference in a multisensory world. J. Neurosci. 39:2301–12
    [Google Scholar]
  188. Zumer JM, White TP, Noppeney U. 2021. The neural mechanisms of audiotactile binding depend on asynchrony. Eur. J. Neurosci. 52:470931
    [Google Scholar]
  189. Zwiers MP, Van Opstal AJ, Paige GD. 2003. Plasticity in human sound localization induced by compressed spatial vision. Nat. Neurosci. 6:175–81
    [Google Scholar]
/content/journals/10.1146/annurev-neuro-100120-085519
Loading
/content/journals/10.1146/annurev-neuro-100120-085519
Loading

Data & Media loading...

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error