1932

Abstract

Surprisal theory proposes that a word's predictability influences processing difficulty because each word requires the comprehender to update a probability distribution over possible sentences. This article first considers the theory's detailed predictions regarding the effects of predictability on reading time and N400 amplitude. Two rather unintuitive predictions appear to be correct based on the current evidence: There is no specific cost when an unpredictable word is encountered in a context where another word is predictable, and the function relating predictability to processing difficulty is logarithmic, not linear. Next, the article addresses the viability of the claim, also associated with Surprisal, that conditional probability is the “causal bottleneck” mediating all effects on incremental processing difficulty. This claim fares less well as conditional probability does not account for the difficulty associated with encountering a low-frequency word or the difficulty associated with garden path disambiguation. Surprisal provides a compelling account of predictability effects but does not provide a complete account of incremental processing difficulty.

Loading

Article metrics loading...

/content/journals/10.1146/annurev-linguistics-011724-121517
2025-02-03
2025-02-08
Loading full text...

Full text loading...

/deliver/fulltext/linguistics/11/1/annurev-linguistics-011724-121517.html?itemId=/content/journals/10.1146/annurev-linguistics-011724-121517&mimeType=html&fmt=ahah

Literature Cited

  1. Abbott MJ, Staub A. 2015.. The effect of plausibility on eye movements in reading: testing EZ Reader's null predictions. . J. Mem. Lang. 85::7687
    [Crossref] [Google Scholar]
  2. Altmann GT, Kamide Y. 1999.. Incremental interpretation at verbs: restricting the domain of subsequent reference. . Cognition 73:(3):24764
    [Crossref] [Google Scholar]
  3. Bemis DK, Pylkkänen L. 2011.. Simple composition: a magnetoencephalography investigation into the comprehension of minimal linguistic phrases. . J. Neurosci. 31:(8):280114
    [Crossref] [Google Scholar]
  4. Berzak Y, Levy R. 2023.. Eye movement traces of linguistic knowledge in native and non-native reading. . Open Mind 7::17996
    [Crossref] [Google Scholar]
  5. Bever TG. 1970.. The Cognitive Basis for Linguistic Structures. New York:: Wiley
    [Google Scholar]
  6. Bornkessel I, Schlesewsky M. 2006.. The extended argument dependency model: a neurocognitive approach to sentence comprehension across languages. . Psychol. Rev. 113:(4):787821
    [Crossref] [Google Scholar]
  7. Brothers T, Kuperberg GR. 2021.. Word predictability effects are linear, not logarithmic: implications for probabilistic models of sentence comprehension. . J. Mem. Lang. 116::104174
    [Crossref] [Google Scholar]
  8. Brothers T, Morgan E, Yacovone A, Kuperberg G. 2023.. Multiple predictions during language comprehension: friends, foes, or indifferent companions?. Cognition 241::105602
    [Crossref] [Google Scholar]
  9. Burnsky J, Kretzschmar F, Mayer E, Staub A. 2023.. The influence of predictability, visual contrast, and preview validity on eye movements and N400 amplitude: co-registration evidence that the N400 reflects late processes. . Lang. Cogn. Neurosci. 38::82142
    [Crossref] [Google Scholar]
  10. Cevoli B, Watkins C, Rastle K. 2022.. Prediction as a basis for skilled reading: insights from modern language models. . R. Soc. Open Sci. 9:(6):211837
    [Crossref] [Google Scholar]
  11. Chang M, Zhang K, Hao L, Zhao S, McGowan VA, et al. 2020.. Word predictability depends on parafoveal preview validity in Chinese reading. . Vis. Cogn. 28:(1):3340
    [Crossref] [Google Scholar]
  12. Clifton C, Staub A. 2011.. Syntactic influences on eye movements during reading. . In The Oxford Handbook of Eye Movements, ed. SP Liversedge, I Gilchrist, S Everling , pp. 896909. Oxford, UK:: Oxford Univ. Press
    [Google Scholar]
  13. Coltheart M, Rastle K, Perry C, Langdon R, Ziegler J. 2001.. DRC: a dual route cascaded model of visual word recognition and reading aloud. . Psychol. Rev. 108:(1):20456
    [Crossref] [Google Scholar]
  14. Dambacher M, Kliegl R, Hofmann M, Jacobs AM. 2006.. Frequency and predictability effects on event-related potentials during reading. . Brain Res. 1084:(1):89103
    [Crossref] [Google Scholar]
  15. Degno F, Loberg O, Zang C, Zhang M, Donnelly N, et al. 2019.. Parafoveal previews and lexical frequency in natural reading: evidence from eye movements and fixation-related potentials. . J. Exp. Psychol. Gen. 148:(3):45374
    [Crossref] [Google Scholar]
  16. DeLong KA, Urbach TP, Kutas M. 2005.. Probabilistic word pre-activation during language comprehension inferred from electrical brain activity. . Nat. Neurosci. 8:(8):111721
    [Crossref] [Google Scholar]
  17. Ehrlich SF, Rayner K. 1981.. Contextual effects on word perception and eye movements during reading. . J. Verbal Learn. Verbal Behav. 20:(6):64155
    [Crossref] [Google Scholar]
  18. Federmeier KD, Kutas M. 1999.. A rose by any other name: long-term memory structure and sentence processing. . J. Mem. Lang. 41:(4):46995
    [Crossref] [Google Scholar]
  19. Federmeier KD, Wlotko EW, De Ochoa-Dewald E, Kutas M. 2007.. Multiple effects of sentential constraint on word processing. . Brain Res. 1146::7584
    [Crossref] [Google Scholar]
  20. Frazier L, Rayner K. 1982.. Making and correcting errors during sentence comprehension: eye movements in the analysis of structurally ambiguous sentences. . Cogn. Psychol. 14:(2):178210
    [Crossref] [Google Scholar]
  21. Frisson S, Harvey DR, Staub A. 2017.. No prediction error cost in reading: evidence from eye movements. . J. Mem. Lang. 95::20014
    [Crossref] [Google Scholar]
  22. Frisson S, Rayner K, Pickering MJ. 2005.. Effects of contextual predictability and transitional probability on eye movements during reading. . J. Exp. Psychol. Learn. Mem. Cogn. 31:(5):86277
    [Crossref] [Google Scholar]
  23. Gibson E. 1998.. Linguistic complexity: locality of syntactic dependencies. . Cognition 68:(1):176
    [Crossref] [Google Scholar]
  24. Gordon PC, Moore M, Choi W, Hoedemaker RS, Lowder MW. 2020.. Individual differences in reading: separable effects of reading experience and processing skill. . Mem. Cogn. 48::55365
    [Crossref] [Google Scholar]
  25. Haeuser KI, Kray J. 2022.. How odd: diverging effects of predictability and plausibility violations on sentence reading and word memory. . Appl. Psycholinguist. 43:(5):1193220
    [Crossref] [Google Scholar]
  26. Hale J. 2001.. A probabilistic Earley parser as a psycholinguistic model. . Proc. NAACL 2::15966
    [Google Scholar]
  27. Hoover JL, Sonderegger M, Piantadosi ST, O'Donnell TJ. 2023.. The plausibility of sampling as an algorithmic theory of sentence processing. . Open Mind 7::35091
    [Google Scholar]
  28. Huang K-J, Arehalli S, Kugemoto M, Muxica C, Prasad G, et al. 2023.. Surprisal does not explain syntactic disambiguation difficulty: evidence from a large-scale benchmark. . PsyArXiv. https://doi.org/10.31234/osf.io/z38u6
  29. Huang K-J, Dillon B. 2023.. An infrequent, large cost underlies garden path effects: an RT distribution approach. Talk presented at the 36th Annual Conference on Human Sentence Processing, Pittsburgh, PA:, Mar. 9–11
    [Google Scholar]
  30. Huettig F. 2015.. Four central questions about prediction in language processing. . Brain Res. 1626::11835
    [Crossref] [Google Scholar]
  31. Huettig F, Pickering MJ. 2019.. Literacy advantages beyond reading: prediction of spoken language. . Trends Cogn. Sci. 23:(6):46475
    [Crossref] [Google Scholar]
  32. Jäger LA, Engelmann F, Vasishth S. 2017.. Similarity-based interference in sentence comprehension: literature review and Bayesian meta-analysis. . J. Mem. Lang. 94::31639
    [Crossref] [Google Scholar]
  33. Keuleers E, Lacey P, Rastle K, Brysbaert M. 2012.. The British Lexicon Project: lexical decision data for 28,730 monosyllabic and disyllabic English words. . Behav. Res. Methods 44::287304
    [Crossref] [Google Scholar]
  34. Kim A, Osterhout L. 2005.. The independence of combinatory semantic processing: evidence from event-related potentials. . J. Mem. Lang. 52:(2):20525
    [Crossref] [Google Scholar]
  35. Kluender R, Kutas M. 1993.. Bridging the gap: evidence from ERPs on the processing of unbounded dependencies. . J. Cogn. Neurosci. 5:(2):196214
    [Crossref] [Google Scholar]
  36. Kretzschmar F, Schlesewsky M, Staub A. 2015.. Dissociating word frequency and predictability effects in reading: evidence from co-registration of eye movements and EEG. . J. Exp. Psychol. Learn. Mem. Cogn. 41:(6):164862
    [Crossref] [Google Scholar]
  37. Kuperberg GR, Jaeger TF. 2016.. What do we mean by prediction in language comprehension?. Lang. Cogn. Neurosci. 31:(1):3259
    [Crossref] [Google Scholar]
  38. Kuperman V, Van Dyke JA. 2011.. Effects of individual differences in verbal skills on eye-movement patterns during sentence reading. . J. Mem. Lang. 65:(1):4273
    [Crossref] [Google Scholar]
  39. Kutas M, Federmeier KD. 2011.. Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP). . Annu. Rev. Psychol. 62::62147
    [Crossref] [Google Scholar]
  40. Kutas M, Hillyard SA. 1984.. Brain potentials during reading reflect word expectancy and semantic association. . Nature 307:(5947):16163
    [Crossref] [Google Scholar]
  41. Kutas M, Van Petten CK, Kluender R. 2006.. Psycholinguistics electrified II (1994–2005). . In Handbook of Psycholinguistics, ed. MJ Traxler, MA Gernsbacher , pp. 659724. London:: Academic. , 2nd ed..
    [Google Scholar]
  42. Levy R. 2008.. Expectation-based syntactic comprehension. . Cognition 106:(3):112677
    [Crossref] [Google Scholar]
  43. Luke SG. 2018.. Influences on and consequences of parafoveal preview in reading. . Attention Percept. Psychophys. 80:(7):167582
    [Crossref] [Google Scholar]
  44. Luke SG, Christianson K. 2016.. Limits on lexical prediction during reading. . Cogn. Psychol. 88::2260
    [Crossref] [Google Scholar]
  45. Marslen-Wilson WD. 1987.. Functional parallelism in spoken word recognition. . Cognition 25:(1–2):71102
    [Crossref] [Google Scholar]
  46. Michaelov JA, Bergen BK. 2023.. Ignoring the alternatives: The N400 is sensitive to stimulus preactivation alone. . Cortex 168::82101
    [Crossref] [Google Scholar]
  47. Morton J. 1969.. Interaction of information in word recognition. . Psychol. Rev. 76:(2):16578
    [Crossref] [Google Scholar]
  48. Ness T, Meltzer-Asscher A. 2018.. Lexical inhibition due to failed prediction: behavioral evidence and ERP correlates. . J. Exp. Psychol. Learn. Mem. Cogn. 44:(8):126985
    [Crossref] [Google Scholar]
  49. Ness T, Meltzer-Asscher A. 2021.. Love thy neighbor: facilitation and inhibition in the competition between parallel predictions. . Cognition 207::104509
    [Crossref] [Google Scholar]
  50. Nieuwland MS, Barr DJ, Bartolozzi F, Busch-Moreno S, Darley E, et al. 2020.. Dissociable effects of prediction and integration during language comprehension: evidence from a large-scale study using brain potentials. . Philos. Trans. R. Soc. B 375:(1791):20180522
    [Crossref] [Google Scholar]
  51. Norris D. 2006.. The Bayesian reader: explaining word recognition as an optimal Bayesian decision process. . Psychol. Rev. 113:(2):32735
    [Crossref] [Google Scholar]
  52. Osterhout L, Holcomb PJ, Swinney DA. 1994.. Brain potentials elicited by garden-path sentences: evidence of the application of verb information during parsing. . J. Exp. Psychol. Learn. Mem. Cogn. 20:(4):786803
    [Crossref] [Google Scholar]
  53. Paape D, Vasishth S. 2022.. Estimating the true cost of garden pathing: a computational model of latent cognitive processes. . Cogn. Sci. 46:(8):e13186
    [Crossref] [Google Scholar]
  54. Perea M, Pollatsek A. 1998.. The effects of neighborhood frequency in reading and lexical decision. . J. Exp. Psychol. Hum. Percept. Perform. 24:(3):76779
    [Crossref] [Google Scholar]
  55. Pickering MJ, Gambi C. 2018.. Predicting while comprehending language: a theory and review. . Psychol. Bull. 144:(10):100244
    [Crossref] [Google Scholar]
  56. Pickering MJ, Garrod S. 2007.. Do people use language production to make predictions during comprehension?. Trends Cogn. Sci. 11:(3):10510
    [Crossref] [Google Scholar]
  57. Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I. 2019.. Language models are unsupervised multitask learners. . OpenAI Blog 1:(8):9
    [Google Scholar]
  58. Rayner K. 1975.. The perceptual span and peripheral cues in reading. . Cogn. Psychol. 16:(1):6581
    [Crossref] [Google Scholar]
  59. Rayner K, Ashby J, Pollatsek A, Reichle ED. 2004a.. The effects of frequency and predictability on eye fixations in reading: implications for the EZ Reader model. . J. Exp. Psychol. Hum. Percept. Perform. 30:(4):72032
    [Crossref] [Google Scholar]
  60. Rayner K, Li X, Juhasz BJ, Yan G. 2005.. The effect of word predictability on the eye movements of Chinese readers. . Psychon. Bull. Rev. 12:(6):108993
    [Crossref] [Google Scholar]
  61. Rayner K, Pacht JM, Duffy SA. 1994.. Effects of prior encounter and global discourse bias on the processing of lexically ambiguous words: evidence from eye fixations. . J. Mem. Lang. 33:(4):52744
    [Crossref] [Google Scholar]
  62. Rayner K, Reichle ED, Stroud MJ, Williams CC. 2006.. The effect of word frequency, word predictability, and font difficulty on the eye movements of young and older readers. . Psychol. Aging 21::44865
    [Crossref] [Google Scholar]
  63. Rayner K, Warren T, Juhasz BJ, Liversedge SP. 2004b.. The effect of plausibility on eye movements in reading. . J. Exp. Psychol. Learn. Mem. Cogn. 30:(6):1290301
    [Crossref] [Google Scholar]
  64. Rayner K, Well AD. 1996.. Effects of contextual constraint on eye movements in reading: a further examination. . Psychon. Bull. Rev. 3::5049
    [Crossref] [Google Scholar]
  65. Roland D, Yun H, Koenig JP, Mauner G. 2012.. Semantic similarity, predictability, and models of sentence processing. . Cognition 122:(3):26779
    [Crossref] [Google Scholar]
  66. Schilling HE, Rayner K, Chumbley JI. 1998.. Comparing naming, lexical decision, and eye fixation times: word frequency effects and individual differences. . Mem. Cogn. 26:(6):127081
    [Crossref] [Google Scholar]
  67. Shain C. 2024.. Word frequency and predictability dissociate in naturalistic reading. . Open Mind 8::177201
    [Crossref] [Google Scholar]
  68. Shain C, Meister C, Pimentel T, Cotterell R, Levy RP. 2024.. Large-scale evidence for logarithmic effects of word predictability on reading time. . PNAS 121:(10):e2307876121
    [Crossref] [Google Scholar]
  69. Slattery TJ. 2009.. Word misperception, the neighbor frequency effect, and the role of sentence context: evidence from eye movements. . J. Exp. Psychol. Hum. Percept. Perform. 35:(6):196975
    [Crossref] [Google Scholar]
  70. Smith NJ, Levy R. 2013.. The effect of word predictability on reading time is logarithmic. . Cognition 128:(3):30219
    [Crossref] [Google Scholar]
  71. Stanovich KE. 1980.. Toward an interactive-compensatory model of individual differences in the development of reading fluency. . Read. Res. Q. 16::3271
    [Crossref] [Google Scholar]
  72. Staub A. 2010.. Eye movements and processing difficulty in object relative clauses. . Cognition 116::7186
    [Crossref] [Google Scholar]
  73. Staub A. 2011a.. The effect of lexical predictability on distributions of eye fixation durations. . Psychon. Bull. Rev. 18::37176
    [Crossref] [Google Scholar]
  74. Staub A. 2011b.. Word recognition and syntactic attachment in reading: evidence for a staged architecture. . J. Exp. Psychol. Gen. 140::40733
    [Crossref] [Google Scholar]
  75. Staub A. 2015.. The effect of lexical predictability on eye movements in reading: critical review and theoretical interpretation. . Lang. Linguist. Compass 9::31127
    [Crossref] [Google Scholar]
  76. Staub A. 2020.. Do effects of visual contrast and font difficulty on readers' eye movements interact with effects of word frequency or predictability?. J. Exp. Psychol. Hum. Percept. Perform. 46::123551
    [Crossref] [Google Scholar]
  77. Staub A. 2021.. How reliable are individual differences in eye movements in reading?. J. Mem. Lang. 116::104190
    [Crossref] [Google Scholar]
  78. Staub A, Goddard K. 2019.. The role of preview validity in predictability and frequency effects on eye movements in reading. . J. Exp. Psychol. Learn. Mem. Cogn. 45::11027
    [Crossref] [Google Scholar]
  79. Staub A, Grant M, Astheimer L, Cohen A. 2015.. The influence of cloze probability and item constraint on cloze task response time. . J. Mem. Lang. 82::117
    [Crossref] [Google Scholar]
  80. Stone K, Nicenboim B, Vasishth S, Rösler F. 2023.. Understanding the effects of constraint and predictability in ERP. . Neurobiol. Lang. 4:(2):22156
    [Crossref] [Google Scholar]
  81. Sturt P. 2003.. The time-course of the application of binding constraints in reference resolution. . J. Mem. Lang. 48:(3):54262
    [Crossref] [Google Scholar]
  82. Swinney DA. 1979.. Lexical access during sentence comprehension: (re)consideration of context effects. . J. Verbal Learn. Verbal Behav. 18:(6):64559
    [Crossref] [Google Scholar]
  83. Szewczyk JM, Federmeier KD. 2022.. Context-based facilitation of semantic access follows both logarithmic and linear functions of stimulus probability. . J. Mem. Lang. 123::104311
    [Crossref] [Google Scholar]
  84. Szewczyk JM, Wodniecka Z. 2020.. The mechanisms of prediction updating that impact the processing of upcoming word: an event-related potential study on sentence comprehension. . J. Exp. Psychol. Learn. Mem. Cogn. 46:(9):171434
    [Crossref] [Google Scholar]
  85. Taylor WL. 1953.. “ Cloze procedure”: a new tool for measuring readability. . Journal. Mass Commun. Q. 30::41533
    [Google Scholar]
  86. Traxler MJ, Pickering MJ, McElree B. 2002.. Coercion in sentence processing: evidence from eye-movements and self-paced reading. . J. Mem. Lang. 47:(4):53047
    [Crossref] [Google Scholar]
  87. Van Berkum JJA, Brown CM, Zwitserlood P, Kooijman V, Hagoort P. 2005.. Anticipating upcoming words in discourse: evidence from ERPs and reading times. . J. Exp. Psychol. Learn. Mem. Cogn. 31::44367
    [Crossref] [Google Scholar]
  88. Van Petten C, Kutas M. 1990.. Interactions between sentence context and word frequency in event-related brain potentials. . Mem. Cogn. 18:(4):38093
    [Crossref] [Google Scholar]
  89. Van Petten C, Luka BJ. 2012.. Prediction during language comprehension: benefits, costs, and ERP components. . Int. J. Psychophysiol. 83:(2):17690
    [Crossref] [Google Scholar]
  90. Van Schijndel M, Linzen T. 2021.. Single-stage prediction models do not explain the magnitude of syntactic disambiguation difficulty. . Cogn. Sci. 45:(6):e12988
    [Crossref] [Google Scholar]
  91. Veldre A, Reichle ED, Wong R, Andrews S. 2020.. The effect of contextual plausibility on word skipping during reading. . Cognition 197::104184
    [Crossref] [Google Scholar]
  92. Von der Malsburg T, Vasishth S. 2011.. What is the scanpath signature of syntactic reanalysis?. J. Mem. Lang. 65:(2):10927
    [Crossref] [Google Scholar]
  93. White SJ, Drieghe D, Liversedge SP, Staub A. 2018.. The word frequency effect during sentence reading: a linear or nonlinear effect of log frequency?. Q. J. Exp. Psychol. 71:(1):4655
    [Crossref] [Google Scholar]
  94. Wicha NYY, Moreno EM, Kutas M. 2004.. Anticipating words and their gender: an event-related brain potential study of semantic integration, gender expectancy, and gender agreement in Spanish sentence reading. . J. Cogn. Neurosci. 16::127288
    [Crossref] [Google Scholar]
  95. Wilcox EG, Pimentel T, Meister C, Cotterell R, Levy RP. 2023.. Testing the predictions of surprisal theory in 11 languages. . Trans. Assoc. Comput. Linguist. 11::145170
    [Crossref] [Google Scholar]
  96. Wong R, Veldre A, Andrews S. 2024.. Are there independent effects of constraint and predictability on eye movements during reading?. J. Exp. Psychol. Learn. Mem. Cogn. 50:(2):33145
    [Crossref] [Google Scholar]
  97. Xu W, Chon J, Liu T, Futrell R. 2023.. The linearity of the effect of surprisal on reading times across languages. . In Findings of the Association for Computational Linguistics: EMNLP 2023, ed. H Bouamor, J Pino, K Bali , pp. 1571121. Stroudsburg, PA:: Assoc. Comput. Linguist.
    [Google Scholar]
/content/journals/10.1146/annurev-linguistics-011724-121517
Loading
  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error