1932
0
  1. Home
  2. A-Z Publications
  3. Annual Review of Psychology
  4. Volume 74, 2023
  5. Article

Annual Review of Psychology

Volume 74, 2023

Dealing with Careless Responding in Survey Data: Prevention, Identification, and Recommended Best Practices

Abstract

Surveys administered online have several benefits, but they are particularly prone to careless responding, which occurs when respondents fail to read item content or give sufficient attention, resulting in raw data that may not accurately reflect respondents’ true levels of the constructs being measured. Careless responding can lead to various psychometric issues, potentially impacting any area of psychology that uses self-reported surveys and assessments. This review synthesizes the careless responding literature to provide a comprehensive understanding of careless responding and ways to prevent, identify, report, and clean careless responding from data sets. Further, we include recommendations for different levels of screening for careless responses. Finally, we highlight some of the most promising areas for future work on careless responding.

Keyword(s): careless responding, insufficient effort responding, invalid responding, lazy respondents, random responding, survey methodology
Loading

Article metrics loading...

/content/journals/10.1146/annurev-psych-040422-045007
2023-01-18
2024-05-20
Loading full text...

Full text loading...

/deliver/fulltext/psych/74/1/annurev-psych-040422-045007.html?itemId=/content/journals/10.1146/annurev-psych-040422-045007&mimeType=html&fmt=ahah

Literature Cited

  1. Anderson A, Ferrell W. 2010. Assessment of qualifications needed by environmental health graduates entering private-sector employment. J. Environ. Health 72:14–20
     [Google Scholar]
  2. Arias V, Garrido L, Jenaro C, Martínez-Molina Arias B. 2020. A little garbage in, lots of garbage out: assessing the impact of careless responding in personality survey data. Behav. Res. Methods 52:62489–505
     [Google Scholar]
  3. Arthur W, Hagen E, George F 2021. The lazy or dishonest respondent: detection and prevention. Annu. Rev. Organ. Psychol. Organ. Behav. 8:105–37
     [Google Scholar]
  4. Baer R, Ballenger J, Berry D, Wetter M. 1997. Detection of random responding on the MMPI–A. J. Pers. Assess. 68:1139–51
     [Google Scholar]
  5. Barber LK, Barnes CM, Carlson KD. 2013. Random and systematic error effects of insomnia on survey behavior. Organ. Res. Methods 16:4616–49
     [Google Scholar]
  6. Barge S, Gehlbach H. 2012. Using the theory of satisficing to evaluate the quality of survey data. Res. High. Educ. 53:2182–200
     [Google Scholar]
  7. Bassett J, Cleveland A, Acorn D, Nix M, Snyder T. 2017. Are they paying attention? Students’ lack of motivation and attention potentially threaten the utility of course evaluations. Assess. Eval. High. Educ. 42:3431–42
     [Google Scholar]
  8. Beach D. 1989. Identifying the random responder. J. Psychol. 123:1101–3
     [Google Scholar]
  9. Beck MF, Albano AD, Smith WM. 2019. Person-fit as an index of inattentive responding: a comparison of methods using polytomous survey data. Appl. Psychol. Meas. 43:5374–87
     [Google Scholar]
  10. Berry D, Wetter M, Baer R, Larsen L, Clark C, Monroe K 1992. MMPI-2 random responding indices: validation using a self-report methodology. Psychol. Assess. 4:3340–45
     [Google Scholar]
  11. Bowling NA, Gibson AM, Houpt JW, Brower CK. 2021. Will the questions ever end? Person-level increases in careless responding during questionnaire completion. Organ. Res. Methods 24:4718–38
     [Google Scholar]
  12. Bowling NA, Huang J, Bragg C, Khazon S, Liu M, Blackmore C. 2016. Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. J. Pers. Soc. Psychol. 111:2218–29
     [Google Scholar]
  13. Brower C. 2018. Too long and too boring: the effects of survey length and interest on careless responding PhD Thesis Wright State Univ. Dayton, OH:
  14. Brower C. 2020. What are you looking at? Using eye-tracking to provide insight into careless responding. PhD Thesis Wright State Univ. Dayton, OH:
     [Google Scholar]
  15. Brühlmann F, Petralito S, Aeschbach L, Opwis K. 2020. The quality of data collected online: an investigation of careless responding in a crowdsourced sample. Psychol. Methods 2:100022
     [Google Scholar]
  16. Camus K. 2015. Once careless, always careless? Temporal and situational stability of insufficient effort responding (IER) PhD Thesis Wright State Univ. Dayton, OH:
  17. Carrier L, Cheever N, Rosen L, Benitez S, Chang J 2009. Multitasking across generations: multitasking choices and difficulty ratings in three generations of Americans. Comput. Hum. Behav. 25:2483–89
     [Google Scholar]
  18. Conijn J, Franz G, Emons W, de Beurs E, Carlier I. 2019. The assessment and impact of careless responding in routine outcome monitoring within mental health care. Multivar. Behav. Res. 54:4593–611
     [Google Scholar]
  19. Curran PG. 2016. Methods for the detection of carelessly invalid responses in survey data. J. Exp. Soc. Psychol. 66:4–19
     [Google Scholar]
  20. Curran PG, Kotrba L, Denison D. 2010. Careless responding in surveys: applying traditional techniques to organizational settings Paper presented at the 25th Annual Conference of the Society for Industrial and Organizational Psychology Atlanta, GA:
  21. Curran PG, Hauser K. 2019. I'm paid biweekly, just not by leprechauns: evaluating valid-but-incorrect response rates to attention check items. J. Res. Pers. 82:103849
     [Google Scholar]
  22. Dunn A, Heggestad E, Shanock L, Theilgard N. 2018. Intra-individual response variability as an indicator of insufficient effort responding: comparison to other indicators and relationships with individual differences. J. Bus. Psychol. 33:1105–21
     [Google Scholar]
  23. Ehlers C, Greene-Shortridge TM, Weekley JA, Zajack MD. 2009. The exploration of statistical methods in detecting random responding Paper presented at the 24th Annual Conference of the Society for Industrial and Organizational Psychology New Orleans, LA:
  24. Francavilla N, Meade A, Young A 2019. Social interaction and internet-based surveys: examining the effects of virtual and in-person proctors on careless response. Appl. Psychol. 68:2223–49
     [Google Scholar]
  25. Furnham A, Hyde G, Trickey G. 2015. Personality and value correlates of careless and erratic questionnaire responses. Pers. Individ. Differ. 80:64–67
     [Google Scholar]
  26. Galesic M, Bosnjak M. 2009. Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opin. Q. 73:2349–60
     [Google Scholar]
  27. Gibson A. 2016. Stop with the questions already! The effects of questionnaire length and monetary incentives on insufficient effort responding Master's Thesis Wright State Univ. Dayton, OH:
  28. Gibson A. 2019. Stop what you're doing, right now! Effects of interactive messages on careless responding PhD Thesis Wright State Univ. Dayton, OH:
  29. Gibson A, Bowling N. 2020. The effects of questionnaire length and behavioral consequences on careless responding. Eur. J. Psychol. Assess. 36:2410–20
     [Google Scholar]
  30. Godinho A, Kushnir V, Cunningham J. 2016. Unfaithful findings: identifying careless responding in addictions research. Addiction 111:6955–56
     [Google Scholar]
  31. Goldammer P, Annen H, Stöckli P, Jonas K 2020. Careless responding in questionnaire measures: detection, impact, and remedies. Leadersh. Q. 31:4101384
     [Google Scholar]
  32. Gough H, Bradley P. 1996. California Psychological Inventory Palo Alto, CA: Consult. Psychol. Press
  33. Haertzen C, Hill H, Belleville R. 1963. Development of the addiction research center inventory (ARCI): selection of items that are sensitive to the effects of various drugs. Psychopharmacologia 4:3155–66
     [Google Scholar]
  34. Huang JL, Bowling N, Liu M, Li Y. 2014. Detecting insufficient effort responding with an infrequency scale: evaluating validity and participant reactions. J. Bus. Psychol. 30:2299–311
     [Google Scholar]
  35. Huang JL, Curran PG, Keeney J, Poposki EM, DeShon RP. 2012. Detecting and deterring insufficient effort responding to surveys. J. Bus. Psychol. 27:199–114
     [Google Scholar]
  36. Huang JL, Liu M, Bowling N. 2015. Insufficient effort responding: examining an insidious confound in survey data. J. Appl. Psychol. 100:3828–45
     [Google Scholar]
  37. Jackson DN. 1976. The appraisal of personal reliability Paper presented at the Meeting of the Society of Multivariate Experimental Psychology University Park, PA:
  38. Johnson J. 2005. Ascertaining the validity of individual protocols from Web-based personality inventories. J. Res. Pers. 39:1103–29
     [Google Scholar]
  39. Jones A, Earnest J, Adam M, Clarke R, Yates J, Pennington C. 2022. Careless responding in crowdsourced alcohol research: a systematic review and meta-analysis of practices and prevalence. Exp. Clin. Psychopharmacol. 30:4381–99
     [Google Scholar]
  40. Kam CCS. 2019. Careless responding threatens factorial analytic results and construct validity of personality measure. Front. Psychol. 10:1258
     [Google Scholar]
  41. Kam CCS, Meyer J. 2015. How careless responding and acquiescence response bias can influence construct dimensionality. Organ. Res. Methods 18:3512–41
     [Google Scholar]
  42. Kurtz J, Parrish C. 2001. Semantic response consistency and protocol validity in structured personality assessment: the case of the NEO-PI-R. . J. Pers. Assess. 76:2315–32
     [Google Scholar]
  43. Maniaci M, Rogge R. 2014. Caring about carelessness: participant inattention and its effects on research. J. Res. Pers. 48:61–83
     [Google Scholar]
  44. Marjanovic Z, Holden R, Struthers W, Cribbie R, Greenglass E. 2015. The inter-item standard deviation (ISD): an index that discriminates between conscientious and random responders. Pers. Individ. 84:79–83
     [Google Scholar]
  45. McGonagle A, Fisher G, Barnes-Farrell J, Grosch J. 2015. Individual and work factors related to perceived work ability and labor force outcomes. J. Appl. Psychol. 100:2376–98
     [Google Scholar]
  46. Meade A, Craig S. 2012. Identifying careless responses in survey data. Psychol. Methods 17:3437–55
     [Google Scholar]
  47. Nichols D, Greene R, Schmolck P. 1989. Criteria for assessing inconsistent patterns of item endorsement on the MMPI: rationale, development, and empirical trials. J. Clin. Psychol. 45:2239–50
     [Google Scholar]
  48. Patton J, Cheng Y, Hong M, Diao Q. 2019. Detection and treatment of careless responses to improve item parameter estimation. J. Educ. Behav. Stat. 44:3309–41
     [Google Scholar]
  49. Paulhus D. 1984. Two-component models of socially desirable responding. J. Pers. Soc. Psychol. 46:3598–609
     [Google Scholar]
  50. Rauti C. 2017. Assessing the effects of survey instructions and physical attractiveness on careless responding in online surveys Master's Thesis Univ. Windsor Windsor, Can:.
  51. Rios J, Guo H, Mao Liu O. 2017. Evaluating the impact of careless responding on aggregated-scores: to filter unmotivated examinees or not?. Int. J. Test. 17:174–104
     [Google Scholar]
  52. Schneider S, May M, Stone A 2018. Careless responding in internet-based quality of life assessments. Qual. Life. Res. 27:41077–88
     [Google Scholar]
  53. Schwarz N. 1999. Self-reports: how the questions shape the answers. Am. Psychol. 54:293–105
     [Google Scholar]
  54. Thompson A. 1975. Random responding and the questionnaire measurement of psychoticism. Soc. Behav. Pers. 3:2111–15
     [Google Scholar]
  55. Ward MK, Meade A. 2018. Applying social psychology to prevent careless responding during online surveys. Appl. Psychol. 67:2231–63
     [Google Scholar]
  56. Ward MK, Meade A, Allred C, Pappalardo G, Stoughton J. 2017. Careless response and attrition as sources of bias in online survey assessments of personality traits and performance. Comput. Hum. Behav. 76:417–30
     [Google Scholar]
  57. Ward MK, Pond S. 2015. Using virtual presence and survey instructions to minimize careless responding on Internet-based surveys. Comput. Hum. Behav. 48:554–68
     [Google Scholar]
  58. Woods C. 2006. Careless responding to reverse-worded items: implications for confirmatory factor analysis. J. Psychopathol. Behav. Assess. 28:3186–91
     [Google Scholar]
  59. Yentes RD. 2020. In search of best practices for the identification and removal of careless responders PhD Thesis N.C. State Univ. Raleigh:
  60. Yentes RD, Wilhelm F. 2018. Careless: procedures for computing indices of careless responding. R Package Version 1.1.3 https://cran.rproject.org/web/packages/careless/index.html
    [Google Scholar]
  61. Zwarun L, Hall A. 2014. What's going on? Age, distraction, and multitasking during online survey taking. Comput. Hum. Behav. 41:236–44
     [Google Scholar]
/content/journals/10.1146/annurev-psych-040422-045007
Loading
Dealing with Careless Responding in Survey Data: Prevention, Identification, and Recommended Best Practices
Annual Review of Psychology 74, 577 (2023); https://doi.org/10.1146/annurev-psych-040422-045007
/content/journals/10.1146/annurev-psych-040422-045007
/content/journals/10.1146/annurev-psych-040422-045007
Loading

Data & Media loading...

  • Article Type: Review Article

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error