1932
0
  1. Home
  2. A-Z Publications
  3. Annual Review of Psychology
  4. Volume 71, 2020
  5. Article

Annual Review of Psychology

Volume 71, 2020

Multisensory Integration as a Window into Orderly and Disrupted Cognition and Communication

Abstract

During our everyday lives, we are confronted with a vast amount of information from several sensory modalities. This multisensory information needs to be appropriately integrated for us to effectively engage with and learn from our world. Research carried out over the last half century has provided new insights into the way such multisensory processing improves human performance and perception; the neurophysiological foundations of multisensory function; the time course for its development; how multisensory abilities differ in clinical populations; and, most recently, the links between multisensory processing and cognitive abilities. This review summarizes the extant literature on multisensory function in typical and atypical circumstances, discusses the implications of the work carried out to date for theory and research, and points toward next steps for advancing the field.

Keyword(s): cognition, development, language, multisensory, review, sensory
Loading

Article metrics loading...

/content/journals/10.1146/annurev-psych-010419-051112
2020-01-04
2024-04-18
Loading full text...

Full text loading...

/deliver/fulltext/psych/71/1/annurev-psych-010419-051112.html?itemId=/content/journals/10.1146/annurev-psych-010419-051112&mimeType=html&fmt=ahah

Literature Cited

  1. Al-azzawi N, Bayram B, Ince G 2018. Audiovisual attention for robots from a developmental perspective. 3rd International Conference on Computer Science and Engineering (UBMK)312–17 New York: IEEE
     [Google Scholar]
  2. Alais D, Burr D. 2004. The ventriloquist effect results from near-optimal bimodal integration. Curr. Biol. 14:257–62
     [Google Scholar]
  3. Am. Psychiatr. Assoc 2013. Diagnostic and Statistical Manual of Mental Disorders Arlington, VA: Am. Psychiatr. Publ. , 5th ed..
  4. Anderson CA, Wiggins IM, Kitterick PT, Hartley DE 2017. Adaptive benefit of cross-modal plasticity following cochlear implantation in deaf adults. PNAS 114:10256–61
     [Google Scholar]
  5. Arsenio AM, Fitzpatrick PM. 2005. Exploiting amodal cues for robot perception. Int. J. Humanoid Robot. 2:125–43
     [Google Scholar]
  6. Bahrick LE. 2010. Intermodal perception and selective attention to intersensory redundancy: implications for typical social development and autism. The Wiley-Blackwell Handbook of Infant Development JG Bremner, TD Wachs 120–66 Malden, MA: Wiley-Blackwell
     [Google Scholar]
  7. Bahrick LE, Flom R, Lickliter R 2002. Intersensory redundancy facilitates discrimination of tempo in 3-month-old infants. Dev. Psychobiol. 41:352–63
     [Google Scholar]
  8. Bahrick LE, Lickliter R. 2000. Intersensory redundancy guides attentional selectivity and perceptual learning in infancy. Dev. Psychol. 36:190–201
     [Google Scholar]
  9. Bahrick LE, Lickliter R. 2002. Intersensory redundancy guides early perceptual and cognitive development. Adv. Child Dev. Behav. 30:153–89
     [Google Scholar]
  10. Bahrick LE, Todd JT. 2012. Multisensory processing in autism spectrum disorders: intersensory processing disturbance as a basis for atypical development. The New Handbook of Multisensory Processes B Stein 657–74 Cambridge, MA: MIT Press
     [Google Scholar]
  11. Bahrick LE, Todd JT, Soska KC 2018. The Multisensory Attention Assessment Protocol (MAAP): characterizing individual differences in multisensory attention skills in infants and children and relations with language and cognition. Dev. Psychol. 54:2207–25
     [Google Scholar]
  12. Baum SH, Stevenson RA, Wallace MT 2015. Behavioral, perceptual, and neural alterations in sensory and multisensory function in autism spectrum disorder. Prog. Neurobiol. 134:140–60
     [Google Scholar]
  13. Bedard G, Barnett-Cowan M. 2016. Impaired timing of audiovisual events in the elderly. Exp. Brain Res. 234:331–40
     [Google Scholar]
  14. Behrendt R-P, Young C. 2004. Hallucinations in schizophrenia, sensory impairment, and brain disease: a unifying model. Behav. Brain Sci. 27:771–87
     [Google Scholar]
  15. Birch HG, Belmont L. 1964. Auditory-visual integration in normal and retarded readers. Am. J. Orthopsychiatry 34:852–61
     [Google Scholar]
  16. Birch HG, Belmont L. 1965. Auditory-visual integration, intelligence and reading ability in school children. Percept. Motor Skills 20:295–305
     [Google Scholar]
  17. Blau V, Reithler J, van Atteveldt N, Seitz J, Gerretsen P et al. 2010. Deviant processing of letters and speech sounds as proximate cause of reading failure: a functional magnetic resonance imaging study of dyslexic children. Brain 133:868–79
     [Google Scholar]
  18. Blau V, van Atteveldt N, Ekkebus M, Goebel R, Blomert L 2009. Reduced neural integration of letters and speech sounds links phonological and reading deficits in adult dyslexia. Curr. Biol. 19:503–8
     [Google Scholar]
  19. Bolognini N, Rasi F, Coccia M, Làdavas E 2005. Visual search improvement in hemianopic patients after audio-visual stimulation. Brain 128:2830–42
     [Google Scholar]
  20. Botvinick M, Cohen J. 1998. Rubber hands ‘feel’ touch that eyes see. Nature 391:756
     [Google Scholar]
  21. Bremner AJ, Lewkowicz DJ, Spence C, eds. 2012a. Multisensory Development Oxford, UK: Oxford Univ. PressRecently published book reviewing typical multisensory development and its disruptions.
  22. Bremner AJ, Lewkowicz DJ, Spence C 2012b. The multisensory approach to development. Multisensory Development AJ Bremner, DJ Lewkowicz, C Spence 1–26 Oxford, UK: Oxford Univ. Press:
     [Google Scholar]
  23. Brown L, Autistic Self Advocacy Netw. 2016. Identity-first language. Autistic Self Advocacy Network http://autisticadvocacy.org/ home/about-asan/identity-first-language/
    [Google Scholar]
  24. Butera IM, Stevenson RA, Mangus BD, Woynaroski TG, Gifford RH, Wallace MT 2018. Audiovisual temporal processing in postlingually deafened adults with cochlear implants. Sci. Rep. 8:11345
     [Google Scholar]
  25. Buzsáki G, Draguhn A. 2004. Neuronal oscillations in cortical networks. Science 304:1926–29
     [Google Scholar]
  26. Campbell J, Sharma A. 2014. Cross-modal re-organization in adults with early stage hearing loss. PLOS ONE 9:e90594
     [Google Scholar]
  27. Cangelosi A, Schlesinger M. 2018. From babies to robots: the contribution of developmental robotics to developmental psychology. Child Dev. Perspect. 12:183–88
     [Google Scholar]
  28. Cappe C, Thelen T, Romei V, Thut G, Murray MM 2012. Looming signals reveal synergistic principles of multisensory integration. J. Neurosci. 32:1171–82
     [Google Scholar]
  29. Carroll JM, Snowling MJ. 2004. Language and phonological skills in children at high risk of reading difficulties. J. Child Psychol. Psychiatry 45:631–40
     [Google Scholar]
  30. Cascio CJ, Foss-Feig JH, Burnette CP, Heacock JL, Cosby AA 2012. The rubber hand illusion in children with autism spectrum disorders: delayed influence of combined tactile and visual input on proprioception. Autism 16:406–19
     [Google Scholar]
  31. Cascio CJ, Woynaroski T, Baranek GT, Wallace M 2016. Toward an interdisciplinary approach to understanding sensory function in autism spectrum disorder. Autism Res 9:920–25
     [Google Scholar]
  32. Chaminade T, Meltzoff AN, Decety J 2005. An fMRI study of imitation: action representation and body schema. Neuropsychologia 43:115–27
     [Google Scholar]
  33. Champoux F, Lepore F, Gagné J-P, Théoret H 2009. Visual stimuli can impair auditory processing in cochlear implant users. Neuropsychologia 47:17–22
     [Google Scholar]
  34. Chan YM, Pianta MJ, McKendrick AM 2014. Older age results in difficulties separating auditory and visual signals in time. J. Vis. 14:13
     [Google Scholar]
  35. Chandrasekaran C, Trubanova A, Stillittano S, Caplier A, Ghazanfar A 2009. The natural statistics of audiovisual speech. PLOS Comput. Biol. 5:e1000436
     [Google Scholar]
  36. Chen SP, Bhattacharya J, Pershing S 2017. Association of vision loss with cognition in older adults. JAMA Ophthalmol 135:963–70
     [Google Scholar]
  37. Choi I, Lee J-Y, Lee S-H 2018. Bottom-up and top-down modulation of multisensory integration. Curr. Opin. Neurobiol. 52:115–22
     [Google Scholar]
  38. Clark A, Chalmers D. 1998. The extended mind. Analysis 58:7–19
     [Google Scholar]
  39. Collignon O, Charbonneau G, Peters F, Nassim M, Lassonde M et al. 2013. Reduced multisensory facilitation in persons with autism. Cortex 49:1704–10
     [Google Scholar]
  40. Cornew L, Roberts TP, Blaskey L, Edgar JC 2012. Resting-state oscillatory activity in autism spectrum disorders. J. Autism Dev. Disord. 42:1884–94
     [Google Scholar]
  41. D'Souza D, D'Souza H, Johnson MH, Karmiloff-Smith A 2016. Audio-visual speech perception in infants and toddlers with Down syndrome, fragile X syndrome, and Williams syndrome. Infant Behav. Dev. 44:249–62
     [Google Scholar]
  42. de Boisferon AH, Tift AH, Minar NJ, Lewkowicz DJ 2018. The redeployment of attention to the mouth of a talking face in the second year of life. J. Exp. Child Psychol. 172:189–200
     [Google Scholar]
  43. de Gelder B, Vroomen J, Van der Heide L 1991. Face recognition and lip-reading in autism. Eur. J. Cogn. Psychol. 3:69–86
     [Google Scholar]
  44. De Niear MA, Gupta PB, Baum SH, Wallace MT 2018. Perceptual training enhances temporal acuity for multisensory speech. Neurobiol. Learn. Mem. 147:9–17
     [Google Scholar]
  45. Dehaene S, Cohen L. 2011. The unique role of the visual word form area in reading. Trends Cogn. Sci. 15:254–62
     [Google Scholar]
  46. Desroches AS, Cone NE, Bolger DJ, Bitan T, Burman DD, Booth JR 2010. Children with reading difficulties show differences in brain regions associated with orthographic processing during spoken language processing. Brain Res 1356:73–84
     [Google Scholar]
  47. Dixon NF, Spitz L. 1980. The detection of auditory visual desynchrony. Perception 9:719–21
     [Google Scholar]
  48. Donohue SE, Darling EF, Mitroff SR 2012. Links between multisensory processing and autism. Exp. Brain Res. 222:377–87
     [Google Scholar]
  49. Eden GF, VanMeter JW, Rumsey JM, Zeffiro TA 1996. The visual deficit theory of developmental dyslexia. Neuroimage 4:S108–17
     [Google Scholar]
  50. Ernst MO, Banks MS. 2002. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415:429–33
     [Google Scholar]
  51. Feldman JI, Dunham K, Cassidy M, Wallace MT, Liu Y, Woynaroski TG. 2018. Audiovisual multisensory integration in individuals with autism spectrum disorder: a systematic review and meta-analysis. Neurosci. Biobehav. Rev 95:220–34Recent review and quantitative synthesis of literature on audiovisual multisensory integration in autism spectrum disorder.
     [Google Scholar]
  52. Festa EK, Katz AP, Ott BR, Tremont G, Heindel WC 2017. Dissociable effects of aging and mild cognitive impairment on bottom-up audiovisual integration. J. Alzheimer's Dis. 59:155–67
     [Google Scholar]
  53. Fine I, Park J-M. 2018. Blindness and human brain plasticity. Annu. Rev. Vis. Sci. 4:337–56
     [Google Scholar]
  54. Fister JK, Stevenson RA, Nidiffer AR, Barnett ZP, Wallace MT 2016. Stimulus intensity modulates multisensory temporal processing. Neuropsychologia 88:92–100
     [Google Scholar]
  55. Flom R, Bahrick LE. 2007. The development of infant discrimination of affect in multimodal and unimodal stimulation: the role of intersensory redundancy. Dev. Psychol. 43:238–52
     [Google Scholar]
  56. Flom R, Bahrick LE. 2010. The effects of intersensory redundancy on attention and memory: infants’ long-term memory for orientation in audiovisual events. Dev. Psychol. 46:428–36
     [Google Scholar]
  57. Foss-Feig JH, Kwakye LD, Cascio CJ, Burnette CP, Kadivar H et al. 2010. An extended multisensory temporal binding window in autism spectrum disorders. Exp. Brain Res. 203:381–89
     [Google Scholar]
  58. Foucher JR, Lacambre M, Pham B-T, Giersch A, Elliott M 2007. Low time resolution in schizophrenia: lengthened windows of simultaneity for visual, auditory and bimodal stimuli. Schizophr. Res. 97:118–27
     [Google Scholar]
  59. Fox E. 1994. Grapheme–phoneme correspondence in dyslexic and matched control readers. Br. J. Psychol. 85:41–53
     [Google Scholar]
  60. Foxe JJ, Molholm S, Del Bene VA, Frey H-P, Russo NN et al. 2013. Severe multisensory speech integration deficits in high-functioning school-aged children with autism spectrum disorder (ASD) and their resolution during early adolescence. Cereb. Cortex 25:298–312
     [Google Scholar]
  61. Francisco AA, Groen MA, Jesse A, McQueen JM 2017. Beyond the usual cognitive suspects: the importance of speechreading and audiovisual temporal sensitivity in reading ability. Learn. Individ. Differ. 54:60–72
     [Google Scholar]
  62. Frank MC, Slemmer JA, Marcus GF, Johnson SP 2009. Information from multiple modalities helps 5‐month‐olds learn abstract rules. Dev. Sci. 12:504–9
     [Google Scholar]
  63. Froyen D, Willems G, Blomert L 2011. Evidence for a specific cross‐modal association deficit in dyslexia: an electrophysiological study of letter–speech sound processing. Dev. Sci. 14:635–48
     [Google Scholar]
  64. Gallese V. 2003. The roots of empathy: the shared manifold hypothesis and the neural basis of intersubjectivity. Psychopathology 36:171–80
     [Google Scholar]
  65. Gascon G, Goodglass H. 1970. Reading retardation and the information content of stimuli in paired associate learning. Cortex 6:417–29
     [Google Scholar]
  66. Gieseler A, Tahden MA, Thiel CM, Colonius H 2018. Does hearing aid use affect audiovisual integration in mild hearing impairment?. Exp. Brain Res. 236:1161–79
     [Google Scholar]
  67. Giraud A-L, Price CJ, Graham JM, Truy E, Frackowiak RS 2001. Cross-modal plasticity underpins language recovery after cochlear implantation. Neuron 30:657–64
     [Google Scholar]
  68. Glazebrook C, Gonzalez D, Hansen S, Elliott D 2009. The role of vision for online control of manual aiming movements in persons with autism spectrum disorders. Autism 13:411–33
     [Google Scholar]
  69. Gogate LJ, Bahrick LE. 1998. Intersensory redundancy facilitates learning of arbitrary relations between vowel sounds and objects in seven-month-old infants. J. Exp. Child Psychol. 69:133–49
     [Google Scholar]
  70. Hahn N, Foxe JJ, Molholm S 2014. Impairments of multisensory integration and cross-sensory learning as pathways to dyslexia. Neurosci. Biobehav. Rev. 47:384–92Fairly recent review on multisensory integration in persons with dyslexia.
     [Google Scholar]
  71. Hairston WD, Burdette JH, Flowers DL, Wood FB, Wallace MT 2005. Altered temporal profile of visual–auditory multisensory interactions in dyslexia. Exp. Brain Res. 166:474–80
     [Google Scholar]
  72. Hämäläinen JA, Salminen HK, Leppänen PH 2013. Basic auditory processing deficits in dyslexia: systematic review of the behavioral and event-related potential/field evidence. J. Learn. Disabil. 46:413–27
     [Google Scholar]
  73. Hancock R, Pugh KR, Hoeft F 2017. Neural noise hypothesis of developmental dyslexia. Trends Cogn. Sci. 21:434–48
     [Google Scholar]
  74. Hari R, Renvall H. 2001. Impaired processing of rapid stimulus sequences in dyslexia. Trends Cogn. Sci. 5:525–32
     [Google Scholar]
  75. Harrar V, Tammam J, Pérez-Bellido A, Pitt A, Stein J, Spence C 2014. Multisensory integration and attention in developmental dyslexia. Curr. Biol. 24:531–35
     [Google Scholar]
  76. Hayes EA, Tiippana K, Nicol TG, Sams M, Kraus N 2003. Integration of heard and seen speech: a factor in learning disabilities in children. Neurosci. Lett. 351:46–50
     [Google Scholar]
  77. Heinrichs DW, Buchanan RW. 1988. Significance and meaning of neurological signs in schizophrenia. Am. J. Psychiatry 145:11–18
     [Google Scholar]
  78. Hillock-Dunn A, Wallace MT. 2012. Developmental changes in the multisensory temporal binding window persist into adolescence. Dev. Sci. 15:688–96
     [Google Scholar]
  79. Huyse A, Berthommier F, Leybaert J 2015. I don't see what you are saying: reduced visual influence on audiovisual speech integration in children with specific language impairment. Phonology 56:22–27
     [Google Scholar]
  80. Insel T, Cuthbert B, Garvey M, Heinssen R, Pine DS et al. 2010. Research domain criteria (RDoC): toward a new classification framework for research on mental disorders. Am. J. Psychiatry 167:748–51
     [Google Scholar]
  81. Irwin JR, Tornatore LA, Brancazio L, Whalen DH 2011. Can children with autism spectrum disorders “hear” a speaking face?. Child Dev 82:1397–403
     [Google Scholar]
  82. Isaiah A, Vongpaisal T, King AJ, Hartley DE 2014. Multisensory training improves auditory spatial processing following bilateral cochlear implantation. J. Neurosci. 34:11119–30
     [Google Scholar]
  83. Jahn KN, Stevenson RA, Wallace MT 2017. Visual temporal acuity is related to auditory speech perception abilities in cochlear implant users. Ear Hearing 38:236–43
     [Google Scholar]
  84. Jaime M, Bahrick L, Lickliter R 2010. The critical role of temporal synchrony in the salience of intersensory redundancy during prenatal development. Infancy 15:61–82
     [Google Scholar]
  85. Jardri R, Pouchet A, Pins D, Thomas P 2011. Cortical activations during auditory verbal hallucinations in schizophrenia: a coordinate-based meta-analysis. Am. J. Psychiatry 168:73–81
     [Google Scholar]
  86. Javitt DC. 2009. Sensory processing in schizophrenia: neither simple nor intact. Schizophr. Bull. 35:1059–64
     [Google Scholar]
  87. Jordan KE, Suanda SH, Brannon EM 2008. Intersensory redundancy accelerates preverbal numerical competence. Cognition 108:210–21
     [Google Scholar]
  88. Kaganovich N, Schumaker J, Leonard LB, Gustafson D, Macias D 2014. Children with a history of SLI show reduced sensitivity to audiovisual temporal asynchrony: an ERP study. J. Speech Lang. Hear. Res. 57:1480–502
     [Google Scholar]
  89. Kanner L. 1943. Autistic disturbances of affective contact. Nerv. Child 2:217–50
     [Google Scholar]
  90. Karipidis II, Pleisch G, Brandeis D, Roth A, Röthlisberger M et al. 2018. Simulating reading acquisition: the link between reading outcome and multimodal brain signatures of letter–speech sound learning in prereaders. Sci. Rep. 8:7121
     [Google Scholar]
  91. Keane BP, Rosenthal O, Chun NH, Shams L 2010. Audiovisual integration in high functioning adults with autism. Res. Autism Spectrum Disord. 4:276–89
     [Google Scholar]
  92. Kéïta L, Mottron L, Dawson M, Bertone A 2011. Atypical lateral connectivity: a neural basis for altered visuospatial processing in autism. Biol. Psychiatry 70:806–11
     [Google Scholar]
  93. Kenny L, Hattersley C, Molins B, Buckley C, Povey C, Pellicano E 2015. Which terms should be used to describe autism? Perspectives from the UK autism community. Autism 20:4442–62
     [Google Scholar]
  94. Kim J-J, Crespo-Facorro B, Andreasen NC, O'Leary DS, Magnotta V, Nopoulos P 2003. Morphology of the lateral superior temporal gyrus in neuroleptic naıïve patients with schizophrenia: relationship to symptoms. Schizophr. Res. 60:173–81
     [Google Scholar]
  95. Kral A, Dorman MF, Wilson BS 2019. Neuronal development of hearing and language: cochlear implants and critical periods. Annu. Rev. Neurosci. 42:47–65
     [Google Scholar]
  96. Lakatos P, Chen C-M, O'Connell MN, Mills A, Schroeder CE 2007. Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53:279–92
     [Google Scholar]
  97. Laurienti PJ, Burdette JH, Maldjian JA, Wallace MT 2006. Enhanced multisensory integration in older adults. Neurobiol. Aging 27:1155–63
     [Google Scholar]
  98. Le Couteur A, Rutter M, Lord C, Rios P, Robertson S et al. 1989. Autism diagnostic interview: a standardized investigator-based instrument. J. Autism Dev. Disord. 19:363–87
     [Google Scholar]
  99. Lewkowicz DJ. 1996. Perception of auditory–visual temporal synchrony in human infants. J. Exp. Psychol. Hum. Percept. Perform. 22:1094
     [Google Scholar]
  100. Lewkowicz DJ. 2004. Perception of serial order in infants. Dev. Sci. 7:175–84
     [Google Scholar]
  101. Lewkowicz DJ, Hansen-Tift AM. 2012. Infants deploy selective attention to the mouth of a talking face when learning speech. PNAS 109:1431–36
     [Google Scholar]
  102. Lickliter R, Bahrick LE, Honeycutt H 2002. Intersensory redundancy facilitates prenatal perceptual learning in bobwhite quail (Colinus virginianus) embryos. Dev. Psychol. 38:15–23
     [Google Scholar]
  103. Lickliter R, Bahrick LE, Honeycutt H 2004. Intersensory redundancy enhances memory in bobwhite quail embryos. Infancy 5:253–69
     [Google Scholar]
  104. Lickliter R, Bahrick LE, Markham RG 2006. Intersensory redundancy educates selective attention in bobwhite quail embryos. Dev. Sci. 9:604–15
     [Google Scholar]
  105. Lickliter R, Banker H. 1994. Prenatal components of intersensory development in precocial birds. The Development of Intersensory Perception: Comparative Perspectives DJ Lewkowicz, R Lickliter 59–80 Hillsdale, NJ: Lawrence Erlbaum Assoc.
     [Google Scholar]
  106. Lin FR, Yaffe K, Xia J, Xue Q-L, Harris TB et al. 2013. Hearing loss and cognitive decline in older adults. JAMA Intern. Med. 173:293–99
     [Google Scholar]
  107. Loughrey DG, Kelly ME, Kelley GA, Brennan S, Lawlor BA 2018. Association of age-related hearing loss with cognitive function, cognitive impairment, and dementia: a systematic review and meta-analysis. JAMA Otolaryngology Head Neck Surg 144:115–26
     [Google Scholar]
  108. Lungarella M, Metta G, Pfeifer R, Sandini G 2003. Developmental robotics: a survey. Connect. Sci. 15:151–90
     [Google Scholar]
  109. Manis FR, Savage PL, Morrison FJ, Horn CC, Howell MJ et al. 1987. Paired associate learning in reading-disabled children: evidence for a rule-learning deficiency. J. Exp. Child Psychol. 43:25–43
     [Google Scholar]
  110. Massaro D. 1998. Perceiving Talking Faces: From Speech Perception to a Behavioral Principle Cambridge, MA: MIT Press
  111. Matson JL 2015. Comorbid Conditions Among Children with Autism Spectrum Disorders Heidelberg, Ger: Springer
  112. McGurk H, MacDonald J. 1976. Hearing lips and seeing voices. Nature 264:746–48Seminal article on one of the most commonly used paradigms of multisensory integration for audiovisual speech.
     [Google Scholar]
  113. Meredith MA, Stein BE. 1986. Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J. Neurophysiol. 56:640–62Seminal article identifying principles of multisensory integration in the cat model.
     [Google Scholar]
  114. Metta G, Fitzpatrick P. 2003. Early integration of vision and manipulation. Adaptive Behav 11:109–28
     [Google Scholar]
  115. Minshew NJ, Sung K, Jones BL, Furman JM 2004. Underdevelopment of the postural control system in autism. Neurology 63:2056–61
     [Google Scholar]
  116. Mittag M, Thesleff P, Laasonen M, Kujala T 2013. The neurophysiological basis of the integration of written and heard syllables in dyslexic adults. Clin. Neurophysiol. 124:315–26
     [Google Scholar]
  117. Mon-Williams MA, Wann JP, Pascal E 1999. Visual–proprioceptive mapping in children with developmental coordination disorder. Dev. Med. Child Neurol. 41:247–54
     [Google Scholar]
  118. Morgan W. 1896. A case of congenital word blindness. Br. Med. J. 2:1378
     [Google Scholar]
  119. Murray MM, Lewkowicz DJ, Amedi A, Wallace MT 2016. Multisensory processes: a balancing act across the lifespan. Trends Neurosci 39:567–79Article reviewing research and theory regarding the nature of multisensory development across the life span.
     [Google Scholar]
  120. Murray MM, Wallace MT. 2011. The Neural Bases of Multisensory Processes Boca Raton, FL: CRC
  121. Nidiffer AR, Stevenson RA, Fister JK, Barnett ZP, Wallace MT 2016. Interactions between space and effectiveness in human multisensory performance. Neuropsychologia 88:83–91
     [Google Scholar]
  122. Norrix LW, Plante E, Vance R 2006. Auditory–visual speech integration by adults with and without language-learning disabilities. J. Commun. Disord. 39:22–36
     [Google Scholar]
  123. Norrix LW, Plante E, Vance R, Boliek CA 2007. Auditory-visual integration for speech by children with and without specific language impairment. J. Speech Lang. Hear. Res. 50:1639–51
     [Google Scholar]
  124. Palmer CJ, Paton B, Hohwy J, Enticott PG 2013. Movement under uncertainty: the effects of the rubber-hand illusion vary along the nonclinical autism spectrum. Neuropsychologia 51:1942–51
     [Google Scholar]
  125. Patten E, Watson LR, Baranek GT 2014. Temporal synchrony detection and associations with language in young children with ASD. Autism Res. Treat. 2014:678346
     [Google Scholar]
  126. Patterson ML, Werker JF. 2003. Two-month-old infants match phonetic information in lips and voice. Dev. Sci. 6:191–96
     [Google Scholar]
  127. Pearl D, Yodashkin-Porat D, Katz N, Valevski A, Aizenberg D et al. 2009. Differences in audiovisual integration, as measured by McGurk phenomenon, among adult and adolescent patients with schizophrenia and age-matched healthy control groups. Compr. Psychiatry 50:186–92
     [Google Scholar]
  128. Peiffer AM, Mozolic JL, Hugenschmidt CE, Laurienti PJ 2007. Age-related multisensory enhancement in a simple audiovisual detection task. Neuroreport 18:1077–81
     [Google Scholar]
  129. Pelham WE, Ross AO. 1977. Selective attention in children with reading problems: a developmental study of incidental learning. J. Abnorm. Child Psychol. 5:1–8
     [Google Scholar]
  130. Pons F, Andreu L, Sanz-Torrent M, Buil-Legaz L, Lewkowicz DJ 2013. Perception of audio-visual speech synchrony in Spanish-speaking children with and without specific language impairment. J. Child Lang. 40:687–700
     [Google Scholar]
  131. Pons F, Sanz‐Torrent M, Ferinu L, Birulés J, Andreu L 2018. Children with SLI can exhibit reduced attention to a talker's mouth. Lang. Learn. 68:180–92
     [Google Scholar]
  132. Postmes L, Sno H, Goedhart S, Van Der Stel J, Heering H, De Haan L 2014. Schizophrenia as a self-disorder due to perceptual incoherence. Schizophr. Res. 152:41–50
     [Google Scholar]
  133. Powers AR, Hevey MA, Wallace MT 2012. Neural correlates of multisensory perceptual learning. J. Neurosci. 32:6263–74
     [Google Scholar]
  134. Powers AR, Hillock AR, Wallace MT 2009. Perceptual training narrows the temporal window of multisensory binding. J. Neurosci. 29:12265–74
     [Google Scholar]
  135. Ramkhalawansingh R, Keshavarz B, Haycock B, Shahab S, Campos JL 2017. Examining the effect of age on visual–vestibular self-motion perception using a driving paradigm. Perception 46:566–85
     [Google Scholar]
  136. Reilly C. 2009. Autism spectrum disorders in Down syndrome: a review. Res. Autism Spectr. Disord. 3:829–39
     [Google Scholar]
  137. Renier L, De Volder AG, Rauschecker JP 2014. Cortical plasticity and preserved function in early blindness. Neurosci. Biobehav. Rev. 41:53–63
     [Google Scholar]
  138. Righi G, Tenenbaum EJ, McCormick C, Blossom M, Amso D, Sheinkopf SJ. 2018. Sensitivity to audio-visual synchrony and its relation to language abilities in children with and without ASD. Autism Res. 11:645–53
     [Google Scholar]
  139. Rohe T, Noppeney U. 2018. Reliability-weighted integration of audiovisual signals can be modulated by top-down attention. eNeuro 5:ENEURO.0315–17 2018.
     [Google Scholar]
  140. Ross LA, Molholm S, Blanco D, Gomez-Ramirez M, Saint-Amour D, Foxe JJ 2011. The development of multisensory speech perception continues into the late childhood years. Eur. J. Neurosci. 33:2329–37
     [Google Scholar]
  141. Ross LA, Saint-Amour D, Leavitt VM, Molholm S, Javitt DC, Foxe JJ 2007. Impaired multisensory processing in schizophrenia: deficits in the visual enhancement of speech comprehension under noisy environmental conditions. Schizophr. Res. 97:173–83
     [Google Scholar]
  142. Rouger J, Lagleyre S, Fraysse B, Deneve S, Deguine O, Barone P 2007. Evidence that cochlear-implanted deaf patients are better multisensory integrators. PNAS 104:7295–300
     [Google Scholar]
  143. Royal DW, Carriere BN, Wallace MT 2009. Spatiotemporal architecture of cortical receptive fields and its impact on multisensory interactions. Exp. Brain Res. 198:127–36
     [Google Scholar]
  144. Rüsseler J, Ye Z, Gerth I, Szycik GR, Münte TF 2018. Audio-visual speech perception in adult readers with dyslexia: an fMRI study. Brain Imaging Behav 12:357–68
     [Google Scholar]
  145. Russo N, Foxe JJ, Brandwein AB, Altschuler T, Gomes H, Molholm S 2010. Multisensory processing in children with autism: high-density electrical mapping of auditory-somatosensory integration. Autism Res 3:253–67
     [Google Scholar]
  146. Salthouse TA. 1996. The processing-speed theory of adult age differences in cognition. Psychol. Rev. 103:403–28
     [Google Scholar]
  147. Sánchez A, Millán-Calenti JC, Lorenzo-López L, Maseda A 2013. Multisensory stimulation for people with dementia: a review of the literature. Am. J. Alzheimer's Dis. Other Dementias 28:7–14
     [Google Scholar]
  148. Sandmann P, Dillier N, Eichele T, Meyer M, Kegel A et al. 2012. Visual activation of auditory cortex reflects maladaptive plasticity in cochlear implant users. Brain 135:555–68
     [Google Scholar]
  149. Schoemaker M, van der Wees M, Flapper B, Verheij-Jansen N, Scholten-Jaegers S, Geuze RH 2001. Perceptual skills of children with developmental coordination disorder. Hum. Mov. Sci. 20:111–33
     [Google Scholar]
  150. Schroeder CE, Lakatos P, Kajikawa Y, Partan S, Puce A 2008. Neuronal oscillations and visual amplification of speech. Trends Cogn. Sci. 12:106–13
     [Google Scholar]
  151. Setti A, Burke KE, Kenny RA, Newell FN 2011. Is inefficient multisensory processing associated with falls in older people?. Exp. Brain Res. 209:375–84
     [Google Scholar]
  152. Setti A, Burke KE, Kenny RA, Newell FN 2013. Susceptibility to a multisensory speech illusion in older persons is driven by perceptual processes. Front. Psychol. 4:575
     [Google Scholar]
  153. Shams L, Kamitani Y, Shimojo S 2000. Illusions: What you see is what you hear. Nature 408:788
     [Google Scholar]
  154. Sharma A, Glick H. 2016. Cross-modal re-organization in clinical populations with hearing loss. Brain Sci 6:4
     [Google Scholar]
  155. Siemann JK, Muller CL, Bamberger G, Allison JD, Veenstra-VanderWeele J, Wallace MT 2015. A novel behavioral paradigm to assess multisensory processing in mice. Front. Behav. Neurosci. 8:456
     [Google Scholar]
  156. Siemann JK, Muller CL, Forsberg CG, Blakely RD, Veenstra-VanderWeele J, Wallace MT 2017. An autism-associated serotonin transporter variant disrupts multisensory processing. Translational Psychiatry 7:e1067
     [Google Scholar]
  157. Simon DM, Wallace MT. 2016. Dysfunction of sensory oscillations in autism spectrum disorder. Neurosci. Biobehav. Rev. 68:848–61
     [Google Scholar]
  158. Sinclair J. 1999. Why I dislike “person-first” language. Autism Mythbusters https://autismmythbusters.com/general-public/autistic-vs-people-with-autism/jim-sinclair-why-i-dislike-person-first-language/
    [Google Scholar]
  159. Spencer KM, Nestor PG, Perlmutter R, Niznikiewicz MA, Klump MC et al. 2004. Neural synchrony indexes disordered perception and cognition in schizophrenia. PNAS 101:17288–93
     [Google Scholar]
  160. Stein BE, Meredith MA. 1993. The Merging of the Senses Cambridge, MA: MIT PressSeminal reference on the nature of multisensory integration.
  161. Stekelenburg JJ, Maes JP, Van Gool AR, Sitskoorn M, Vroomen J 2013. Deficient multisensory integration in schizophrenia: an event-related potential study. Schizophr. Res. 147:253–61
     [Google Scholar]
  162. Stevenson RA, Baum SH, Krueger J, Newhouse PA, Wallace MT 2018a. Links between temporal acuity and multisensory integration across life span. J. Exp. Psychol. Hum. Percept. Perform. 44:106–16
     [Google Scholar]
  163. Stevenson RA, Baum SH, Segers M, Ferber S, Barense MD, Wallace MT 2017a. Multisensory speech perception in autism spectrum disorder: from phoneme to whole‐word perception. Autism Res 10:1280–90
     [Google Scholar]
  164. Stevenson RA, Nelms CE, Baum SH, Zurkovsky L, Barense MD et al. 2015. Deficits in audiovisual speech perception in normal aging emerge at the level of whole-word recognition. Neurobiol. Aging 36:283–91
     [Google Scholar]
  165. Stevenson RA, Park S, Cochran C, McIntosh LG, Noel J-P et al. 2017b. The associations between multisensory temporal processing and symptoms of schizophrenia. Schizophr. Res. 179:97–103
     [Google Scholar]
  166. Stevenson RA, Segers M, Ferber S, Barense MD, Camarata S, Wallace MT 2016. Keeping time in the brain: autism spectrum disorder and audiovisual temporal processing. Autism Res 9:720–38
     [Google Scholar]
  167. Stevenson RA, Segers M, Ncube BL, Black KR, Bebko JM et al. 2018b. The cascading influence of multisensory processing on speech perception in autism. Autism 22:609–24
     [Google Scholar]
  168. Stevenson RA, Sheffield SW, Butera IM, Gifford RH, Wallace MT 2017c. Multisensory integration in cochlear implant recipients. Ear Hear 38:521–38
     [Google Scholar]
  169. Stevenson RA, Siemann JK, Woynaroski TG, Schneider BC, Eberly HE et al. 2014a. Brief report: arrested development of audiovisual speech perception in autism spectrum disorders. J. Autism Dev. Disord. 44:1470–77
     [Google Scholar]
  170. Stevenson RA, Siemann JK, Woynaroski TG, Schneider BC, Eberly HE et al. 2014b. Evidence for diminished multisensory integration in autism spectrum disorders. J. Autism Dev. Disord. 44:3161–67
     [Google Scholar]
  171. Stevenson RA, Zemtsov RK, Wallace MT 2012. Individual differences in the multisensory temporal binding window predict susceptibility to audiovisual illusions. J. Exp. Psychol. Hum. Percept. Perform. 38:1517–29
     [Google Scholar]
  172. Stewart ME, Ota M. 2008. Lexical effects on speech perception in individuals with “autistic” traits. Cognition 109:157–62
     [Google Scholar]
  173. Strelnikov K, Rouger J, Lagleyre S, Fraysse B, Démonet JF et al. 2015. Increased audiovisual integration in cochlear‐implanted deaf patients: independent components analysis of longitudinal positron emission tomography data. Eur. J. Neurosci. 41:677–85
     [Google Scholar]
  174. Striem-Amit E, Cohen L, Dehaene S, Amedi A 2012. Reading with sounds: sensory substitution selectively activates the visual word form area in the blind. Neuron 76:640–52
     [Google Scholar]
  175. Sumby WH, Pollack I. 1954. Visual contribution to speech intelligibility in noise. J. Acoust. Soc. Am. 26:212–15
     [Google Scholar]
  176. Surguladze SA, Calvert GA, Brammer MJ, Campbell R, Bullmore ET et al. 2001. Audio–visual speech perception in schizophrenia: an fMRI study. Psychiatry Res. Neuroimaging 106:1–14
     [Google Scholar]
  177. Szycik GR, Münte TF, Dillo W, Mohammadi B, Samii A et al. 2009. Audiovisual integration of speech is disturbed in schizophrenia: an fMRI study. Schizophr. Res. 110:111–18
     [Google Scholar]
  178. Targher S, Micciolo R, Occelli V, Zampini M 2017. The role of temporal disparity on audiovisual integration in low-vision individuals. Perception 46:1356–70
     [Google Scholar]
  179. Targher S, Occelli V, Zampini M 2012. Audiovisual integration in low vision individuals. Neuropsychologia 50:576–82
     [Google Scholar]
  180. Taylor N, Isaac C, Milne E 2010. A comparison of the development of audiovisual integration in children with autism spectrum disorders and typically developing children. J. Autism Dev. Disord. 40:1403–11
     [Google Scholar]
  181. Teinonen T, Aslin RN, Alku P, Csibra G 2008. Visual speech contributes to phonetic learning in 6-month-old infants. Cognition 108:850–55
     [Google Scholar]
  182. Ten Oever S, Sack AT, Wheat KL, Bien N, Van Atteveldt N 2013. Audio-visual onset differences are used to determine syllable identity for ambiguous audio-visual stimulus pairs. Front. Psychol. 4:331
     [Google Scholar]
  183. Tenenbaum EJ, Sobel DM, Sheinkopf SJ, Shah RJ, Malle BF, Morgan JL 2015. Attention to the mouth and gaze following in infancy predict language development. J. Child Lang. 42:1173–90
     [Google Scholar]
  184. Thompson E. 2005. Sensorimotor subjectivity and the enactive approach to experience. Phenomenol. Cogn. Sci. 4:407–27
     [Google Scholar]
  185. Torres-Jara E, Natale L, Fitzpatrick P 2005. Tapping into touch. Proceedings of the Fifth International Workshop on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems L Berthouze 79–86 Lund, Swed: LUCS
     [Google Scholar]
  186. Tye-Murray N, Sommers M, Spehar B, Myerson J, Hale S 2010. Aging, audiovisual integration, and the principle of inverse effectiveness. Ear Hear 31:636–44
     [Google Scholar]
  187. Tye-Murray N, Spehar B, Myerson J, Hale S, Sommers M 2016. Lipreading and audiovisual speech recognition across the adult lifespan: implications for audiovisual integration. Psychol. Aging 31:380–89
     [Google Scholar]
  188. Uhlhaas PJ, Singer W. 2010. Abnormal neural oscillations and synchrony in schizophrenia. Nat. Rev. Neurosci. 11:100–13
     [Google Scholar]
  189. Ujiie Y, Asai T, Wakabayashi A 2015. The relationship between level of autistic traits and local bias in the context of the McGurk effect. Front. Psychol. 6:891
     [Google Scholar]
  190. van der Smagt MJ, van Engeland H, Kemner C 2007. Brief report: Can you see what is not there? Low-level auditory-visual integration in autism spectrum disorder. J. Autism Dev. Disord. 37:2014–19
     [Google Scholar]
  191. van Laarhoven T, Keetels M, Schakel L, Vroomen J 2018. Audio‐visual speech in noise perception in dyslexia. Dev. Sci. 21:e12504
     [Google Scholar]
  192. Varela F, Thompson E, Rosch E 1991. The Embodied Mind: Cognitive Science and Human Experience Cambridge, MA: MIT Press
  193. Visser J. 2003. Developmental coordination disorder: a review of research on subtypes and comorbidities. Hum. Mov. Sci. 22:479–93
     [Google Scholar]
  194. Vroomen J, Keetels M. 2010. Perception of intersensory synchrony: a tutorial review. Atten. Percept. Psychophys. 72:871–84
     [Google Scholar]
  195. Wagner RK. 1986. Phonological processing abilities and reading: implications for disabled readers. J. Learn. Disabil. 19:623–29
     [Google Scholar]
  196. Wallace MT, Stein BE. 1997. Development of multisensory neurons and multisensory integration in cat superior colliculus. J. Neurosci. 17:2429–44
     [Google Scholar]
  197. Wallace MT, Stein BE. 2001. Sensory and multisensory responses in the newborn monkey superior colliculus. J. Neurosci. 21:8886–94
     [Google Scholar]
  198. Wallace MT, Stevenson RA. 2014. The construct of the multisensory temporal binding window and its dysregulation in developmental disabilities. Neuropsychologia 64:105–23Fairly recent review of findings for multisensory temporal processing in individuals with developmental disabilities.
     [Google Scholar]
  199. Wang D, Xin J. 2018. Emergent spatio-temporal multimodal learning using a developmental network. Appl. Intell. 49:1306–23
     [Google Scholar]
  200. Williams LE, Light GA, Braff DL, Ramachandran VS 2010. Reduced multisensory integration in patients with schizophrenia on a target detection task. Neuropsychologia 48:3128–36
     [Google Scholar]
  201. Wu J, Yang J, Yu Y, Li Q, Nakamura N et al. 2012. Delayed audiovisual integration of patients with mild cognitive impairment and Alzheimer's disease compared with normal aged controls. J. Alzheimer's Dis. 32:317–28
     [Google Scholar]
  202. Yaguchi A, Hidaka S. 2018. Distinct autistic traits are differentially associated with the width of the multisensory temporal binding window. Multisensory Res 31:523–36
     [Google Scholar]
  203. Yu AC, Grove J, Martinovic M, Sonderegger M 2011. Effects of working memory capacity and “autistic” traits on phonotactic effects in speech perception. Proceedings of the 17th International Congress of Phonetic Sciences (ICPhS XVII): August 17–21, 20112236–39 Hong Kong: City Univ. Hong Kong
     [Google Scholar]
  204. Zhang J, Meng Y, He J, Xiang Y, Wu C et al. 2019. McGurk effect by individuals with autism spectrum disorder and typically developing controls: a systematic review and meta-analysis. J. Autism Dev. Disord. 49:34–43
     [Google Scholar]
  205. Zhang Y, Weng J. 2010. Spatio–temporal multimodal developmental learning. IEEE Trans. Autonomous Ment. Dev. 2:149–66
     [Google Scholar]
  206. Ziegler JC, Pech‐Georgel C, George F, Lorenzi C 2009. Speech‐perception‐in‐noise deficits in dyslexia. Dev. Sci. 12:732–45
     [Google Scholar]
  207. Zurif E, Carson G. 1970. Dyslexia in relation to cerebral dominance and temporal analysis. Neuropsychologia 8:351–61
     [Google Scholar]
/content/journals/10.1146/annurev-psych-010419-051112
Loading
  • Article Type: Review Article

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error