1932
0
  1. Home
  2. A-Z Publications
  3. Annual Review of Organizational Psychology and Organizational Behavior
  4. Volume 4, 2017
  5. Article

Annual Review of Organizational Psychology and Organizational Behavior

Volume 4, 2017

Retaking Employment Tests: What We Know and What We Still Need to Know

Abstract

The extent to which test scores change upon retesting has important implications for both organizations and individuals who apply to those organizations. We review research on retesting and score changes that dates back nearly 100 years. Our findings suggest that compared to initial test scores, retest scores tend to be higher, more varied, and more reliable and tend to demonstrate somewhat stronger relations with criteria such as academic and job performance. There also is some evidence that retesting can change the constructs test scores reflect. However, empirical research has yet to clearly delineate factors that underlie such differences between initial and retest scores. We discuss implications of these findings for organizations and applicants. We also identify key unanswered questions about retesting that future research should address.

Keyword(s): personnel, practice effects, retest effects, retesting, score changes, testing effect
Loading

Article metrics loading...

/content/journals/10.1146/annurev-orgpsych-032516-113349
2017-03-21
2024-04-20
Loading full text...

Full text loading...

/deliver/fulltext/orgpsych/4/1/annurev-orgpsych-032516-113349.html?itemId=/content/journals/10.1146/annurev-orgpsych-032516-113349&mimeType=html&fmt=ahah

Literature Cited

  1. Allalouf A, Ben‐Shakhar G. 1998. The effect of coaching on the predictive validity of scholastic aptitude tests. J. Educ. Meas. 35:131–47 [Google Scholar]
  2. Alliger GM, Katzman S. 1997. When training affects variability: beyond the assessment of mean differences in training evaluation. Improving Training Effectiveness in Work Organizations JK Ford 223–43 Mahwah, NJ: Erlbaum [Google Scholar]
  3. Am. Educ. Res. Assoc. (AERA), Am. Psychol. Assoc. (APA), Nat. Council Meas. Educ. (NCME) 1999. Standards for Educational and Psychological Testing Washington, DC: AERA
  4. Appelrouth JI, Zabrucky KM, Moore D. 2015. Preparing students for college admissions tests. Assess. Educ. Principles Policy Pract. 2015:1–18 [Google Scholar]
  5. Arthur W, Glaze RM, Villado AJ, Taylor JE. 2010. The magnitude and extent of cheating and response distortion effects on unproctored internet‐based tests of cognitive ability and personality. Int. J. Sel. Assess. 18:11–16 [Google Scholar]
  6. Arvey RD, Sackett PR. 1993. Fairness in selection: current developments and perspectives. Personnel Selection in Organizations N Schmitt, WC Borman 171–202 San Francisco: Jossey-Bass [Google Scholar]
  7. Bobko P. 2001. Correlation and Regression: Applications for Industrial and Organizational Psychology and Management Thousand Oaks, CA: Sage, 2nd ed..
  8. Brogden HE. 1949. When testing pays off. Pers. Psychol. 2:2171–83 [Google Scholar]
  9. Burke EF. 1997. A short note on the persistence of retest effects on aptitude scores. J. Occup. Organ. Psychol. 70:3295–301 [Google Scholar]
  10. Butcher JN, Morfitt RC, Rouse SV, Holden RR. 1997. Reducing MMPI-2 defensiveness: the effect of specialized instructions on retest validity in a job applicant sample. J. Pers. Assess. 68:2385–401 [Google Scholar]
  11. Campbell DT, Stanley JC. 1963. Experimental and Quasi-Experimental Designs for Research Chicago: Rand McNally
  12. Catano VM, Brochu A, Lamerson CD. 2012. Assessing the reliability of situational judgment tests used in high‐stakes situations. Int. J. Sel. Assess. 20:3333–46 [Google Scholar]
  13. Cigrang JA, Staal MA. 2001. Readministration of the MMPI-2 following defensive invalidation in a military job applicant sample. J. Pers. Assess. 76:3472–81 [Google Scholar]
  14. Cohen J. 1988. Statistical Power Analysis for the Behavioral Sciences Hillsdale, NJ: Erlbaum, 2nd ed..
  15. Costa PT, McCrae RR. 1988. Personality in adulthood: a six-year longitudinal study of self-reports and spouse ratings on the NEO Personality Inventory. J. Pers. Soc. Psychol. 54:5853–63 [Google Scholar]
  16. Coyle TR. 2006. Test–retest changes on scholastic aptitude tests are not related to g. Intelligence 34:115–27 [Google Scholar]
  17. Donovan JJ, Dwight SA, Schneider D. 2014. The impact of applicant faking on selection measures, hiring decisions, and employee performance. J. Bus. Psychol. 29:3479–93 [Google Scholar]
  18. Dunlap K, Snyder A. 1920. Practice effects in intelligence tests. J. Exp. Psychol. 3:5396–403 [Google Scholar]
  19. Dunlop PD, Morrison DL, Cordery JL. 2011. Investigating retesting effects in a personnel selection context. Int. J. Sel. Assess. 19:2217–21 [Google Scholar]
  20. Dweck CS. 1986. Motivational processes affecting learning. Am. Psychol. 41:101040–48 [Google Scholar]
  21. Ellingson JE, Heggestad ED, Makarius EE. 2012. Personality retesting for managing intentional distortion. J. Pers. Soc. Psychol. 102:51063–76 [Google Scholar]
  22. Ellingson JE, Sackett PR, Connelly BS. 2007. Personality assessment across selection and development contexts: insights into response distortion. J. Appl. Psychol. 92:2386–95 [Google Scholar]
  23. Feinberg RA, Raymond MR, Haist SA. 2015. Repeat testing effects on credentialing exams: Are repeaters misinformed or uninformed. Educ. Meas. Issues Practice 34:134–39 [Google Scholar]
  24. Ferguson LW. 1943. The effects of a second administration of an employment test. J. Appl. Psychol. 27:2170–75 [Google Scholar]
  25. Ferrando PJ. 2003. Analyzing retest increases in reliability: a covariance structure modeling approach. Struct. Equation Model. 10:2222–37 [Google Scholar]
  26. Fleishman EA, Hempel WE Jr. 1954. Changes in factor structure of a complex psychomotor test as a function of practice. Psychometrika 19:3239–52 [Google Scholar]
  27. Galton F. 1886. Regression towards mediocrity in hereditary stature. J. Anthropol. Inst. 15:246–63 [Google Scholar]
  28. Geving AM, Webb S, Davis B. 2005. Opportunities for repeat testing: Practice doesn't always make perfect. Appl. HRM Res. 10:247–56 [Google Scholar]
  29. Gilliland SW. 1993. The perceived fairness of selection systems: an organizational justice perspective. Acad. Manag. Rev. 18:694–734 [Google Scholar]
  30. Gilmore ME. 1927. Coaching for intelligence tests. J. Educ. Psychol. 18:2119–21 [Google Scholar]
  31. Gordon LV, Stapleton ES. 1956. Fakability of a forced-choice personality test under realistic high school employment conditions. J. Appl. Psychol. 40:4258–62 [Google Scholar]
  32. Gordon ME, Cohen SL. 1973. Training behavior as a predictor of trainability. Pers. Psychol. 26:2261–72 [Google Scholar]
  33. Griffith RL, Chmielowski T, Yoshita Y. 2007. Do applicants fake? An examination of the frequency of applicant faking behavior. Pers. Rev. 36:3341–55 [Google Scholar]
  34. Hausknecht JP. 2010. Candidate persistence and personality test practice effects: implications for staffing system management. Pers. Psychol. 63:299–324 [Google Scholar]
  35. Hausknecht JP, Day DV, Thomas SC. 2004. Applicant reactions to selection procedures: an updated model and meta-analysis. Pers. Psychol. 57:639–83 [Google Scholar]
  36. Hausknecht JP, Halpert JA, Di Paolo NT, Moriarty Gerrard MO. 2007. Retesting in selection: a meta-analysis of practice effects for tests of cognitive ability. J. Appl. Psychol. 92:373–85 [Google Scholar]
  37. Hausknecht JP, Trevor CO, Farr JL. 2002. Retaking ability tests in a selection setting: implications for practice effects, training performance, and turnover. J. Appl. Psychol. 87:2243–54 [Google Scholar]
  38. Hermes M, Stelling D. 2016. Context matters, but how much? Latent state–trait analysis of cognitive ability assessments. Int. J. Sel. Assess. 24:3285–95 [Google Scholar]
  39. Hogan J, Barrett P, Hogan R. 2007. Personality measurement, faking, and employment selection. J. Appl. Psychol. 92:1270–85 [Google Scholar]
  40. Holladay CL, David E, Johnson SK. 2013. Retesting personality in employee selection: Implications of the context, sample, and setting. Psychol. Rep. 112:2486–501 [Google Scholar]
  41. Horton AM. 1992. Neuropsychological practice effects × age: A brief note. Percept. Motor Skills 75:1257–58 [Google Scholar]
  42. House RJ, Hanges PJ, Javidan M, Dorfman PW, Gupta V. 2004. Culture, Leadership, and Organizations: The GLOBE Study of 62 Societies Thousand Oaks, CA: Sage
  43. Howard KI. 1964. Differentiation of individuals as a function of repeated testing. Educ. Psychol. Meas. 24:2875–94 [Google Scholar]
  44. Ingold PV, Kleinmann M, König CJ, Melchers KG. 2016. Transparency of assessment centers: Lower criterion-related validity but greater opportunity to perform. Pers. Psychol. 69:2467–97 [Google Scholar]
  45. Jagodzinski W, Kühnel SM, Schmidt P. 1987. Is there a “Socratic effect” in nonexperimental panel studies? Consistency of an attitude toward guestworkers. Sociol. Methods Res. 15:3259–302 [Google Scholar]
  46. Jansen A, Melchers KG, Lievens F, Kleinmann M, Brändli M. et al. 2013. Situation assessment as an ignored factor in the behavioral consistency paradigm underlying the validity of personnel selection procedures. J. Appl. Psychol. 98:2326–41 [Google Scholar]
  47. Jensen AR. 1998. The g Factor: The Science of Mental Ability Westport, CT: Praeger
  48. Kelley PL, Jacobs RR, Farr JL. 1994. Effects of multiple administrations of the MMPI for employee screening. Pers. Psychol. 47:3575–91 [Google Scholar]
  49. Kleinmann M, Ingold PV, Lievens F, Jansen A, Melchers KG, König CJ. 2011. A different look at why selection procedures work: the role of candidates’ ability to identify criteria. Organ. Psychol. Rev. 1:2128–46 [Google Scholar]
  50. Knowles ES, Coker MC, Scott RA, Cook DA, Neville JW. 1996. Measurement-induced improvement in anxiety: mean shifts with repeated assessment. J. Pers. Soc. Psychol. 71:2352–63 [Google Scholar]
  51. Kulik JA, Kulik CLC, Bangert RL. 1984. Effects of practice on aptitude and achievement test scores. Am. Educ. Res. J. 21:2435–47 [Google Scholar]
  52. LaHuis DM, MacLane CN, Schlessman BR. 2007. Do applicants' perceptions matter? Investigating reapplication behavior using fairness theory. Int. J. Sel. Assess. 15:4383–93 [Google Scholar]
  53. Landers RN, Sackett PR, Tuzinski KA. 2011. Retesting after initial failure, coaching rumors, and warnings against faking in online personality measures for selection. J. Appl. Psychol. 96:2002–10 [Google Scholar]
  54. Lievens F, Buyse T, Sackett PR. 2005. Retesting effects in operational selection settings: development and test of a framework. Pers. Psychol. 58:981–1007 [Google Scholar]
  55. Lievens F, Reeve CL, Heggestad ED. 2007. An examination of psychometric bias due to retesting on cognitive ability tests in selection settings. J. Appl. Psychol. 92:1672–82 [Google Scholar]
  56. Leventhal GS. 1980. What should be done with equity theory? New approaches to the study of fairness in social relationships. Social Exchange: Advances in Theory and Research K Gergen, M Greenberg, R Willis 27–55 New York: Plenum Press [Google Scholar]
  57. Matton N, Vautier S, Raufaste E. 2009. Situational effects may account for gain scores in cognitive ability testing: a longitudinal SEM approach. Intelligence 37:4412–21 [Google Scholar]
  58. Matton N, Vautier S, Raufaste E. 2011. Test‐specificity of the advantage of retaking cognitive ability tests. Int. J. Sel. Assess. 19:111–17 [Google Scholar]
  59. McCarthy JM, Van Iddekinge CH, Lievens F, Kung MC, Sinar EF, Campion MA. 2013. Do candidate reactions relate to job performance or affect criterion-related validity? A multistudy investigation of relations among reactions, selection test scores, and job performance. J. Appl. Psychol. 98:5701–19 [Google Scholar]
  60. McGuire WJ. 1960. Cognitive consistency and attitude change. J. Abnorm. Soc. Psychol. 60:3345–54 [Google Scholar]
  61. McIntyre F. 1980. The reliability of assessment center results after feedback. J. Assess. Cent. Technol. 3:110–14 [Google Scholar]
  62. Meng XL, Rosenthal R, Rubin DB. 1992. Comparing correlated correlation coefficients. Psychol. Bull. 111:1172–75 [Google Scholar]
  63. Millman J, Bishop CH, Ebel R. 1965. An analysis of test-wiseness. Educ. Psychol. Meas. 25:3707–26 [Google Scholar]
  64. Patterson BF, Mattern KD, Swerdzewski P. 2012. Are the best scores the best scores for predicting college success. J. Coll. Admiss. 217:34–45 [Google Scholar]
  65. Ployhart RE, Harold CM. 2004. The Applicant Attribution-Reaction Theory (AART): an integrative theory of applicant attributional processing. Int. J. Sel. Assess. 12:84–98 [Google Scholar]
  66. Rapport LJ, Brines DB, Axelrod BN, Theisen ME. 1997. Full scale IQ as mediator of practice effects: The rich get richer. Clin. Neuropsychol. 11:375–80 [Google Scholar]
  67. Raymond MR, Neustel S, Anderson D. 2007. Retest effects on identical and parallel forms in certification and licensure testing. Pers. Psychol. 60:367–96 [Google Scholar]
  68. Reeve CL, Lam H. 2005. The psychometric paradox of practice effects due to retesting: measurement invariance and stable ability estimates in the face of observed score changes. Intelligence 33:5535–49 [Google Scholar]
  69. Reeve CL, Lam H. 2007. The relation between practice effects, test-taker characteristics and degree of g-saturation. Int. J. Test. 7:225–42 [Google Scholar]
  70. Renshaw S. 1923. The intelligence of teachers in training, with a note on practice-effects with intelligence tests. J. Educ. Res. 7:128–36 [Google Scholar]
  71. Richardson F, Robinson ES. 1921. Effects of practice upon the scores and predictive value of the alpha intelligence examination. J. Exp. Psychol. 4:4300–17 [Google Scholar]
  72. Roediger HL III, Karpicke JD. 2006. The power of testing memory: basic research and implications for educational practice. Perspect. Psychol. Sci. 1:181–210 [Google Scholar]
  73. Ryan AM, Ployhart RE. 2000. Applicants’ perceptions of selection procedures and decisions: a critical review and agenda for the future. J. Manag. 26:3565–606 [Google Scholar]
  74. Sackett PR, Burris LR, Ryan AM. 1989. Coaching and practice effects in personnel selection. International Review of Industrial and Organizational Psychology CL Cooper, IT Robertson 145–83 Oxford, UK: Wiley [Google Scholar]
  75. Schinkel S, van Dierendonck D, van Vianen A, Ryan AM. 2011. Applicant reactions to rejection. J. Pers. Psychol. 10:4146–56 [Google Scholar]
  76. Schleicher DJ, Van Iddekinge CH, Morgeson FP, Campion MA. 2010. If at first you don't succeed, try, try again: understanding race, gender, and age differences in selection test score improvement. J. Appl. Psychol. 95:603–17 [Google Scholar]
  77. Schmidt FL, Hunter JE. 1998. The validity and utility of selection methods in personnel psychology: practical and theoretical implications of 85 years of research findings. Psychol. Bull. 124:2262–74 [Google Scholar]
  78. Sin HP, Farr JL, Murphy KR, Hausknecht JP. 2004. An investigation of Black-White differences in self-selection and performance in repeated testing Presented at Annu. Meet. Acad. Manag., 64th, New Orleans, Louisiana
  79. Soc. Ind. Org. Psychol. (SIOP). 2003. Principles for the Validation and Use of Personnel Selection Procedures Bowling Green, OH: SIOP, 4th ed..
  80. te Nijenhuis J, van Vianen AE, van der Flier H. 2007. Score gains on g-loaded tests: no g. Intelligence 35:3283–300 [Google Scholar]
  81. The College Board. 2012. SAT score-use practices by participating institution Rep., Coll. Board., New York. http://professionals.collegeboard.com/profdownload/sat-score-use-practices-list.pdf
  82. Thorndike EL. 1908. The effect of practice in the case of a purely intellectual function. Am. J. Psychol. 19:3374–84 [Google Scholar]
  83. Thorndike EL. 1922. Practice effects in intelligence tests. J. Exp. Psychol. 5:2101–7 [Google Scholar]
  84. Thorndike RL. 1949. Personnel Selection New York: Wiley
  85. Topp BW. 2011. An exploratory study of two decades of promotional testing in a metropolitan police department. J. Police Crim. Psychol. 26:2143–51 [Google Scholar]
  86. Van Iddekinge CH, Morgeson FP, Schleicher DJ, Campion MA. 2011. Can I retake it? Exploring subgroup differences and criterion-related validity in promotion retesting. J. Appl. Psychol. 96:941–55 [Google Scholar]
  87. Villado AJ, Randall JG, Zimmer CU. 2016. The effect of method characteristics on retest score gains and criterion-related validity. J. Bus. Psychol. 31:2233–48 [Google Scholar]
  88. Walfish S. 2007. Reducing Minnesota Multiphasic Personality Inventory defensiveness: effect of specialized instructions on retest validity in a sample of preoperative bariatric patients. Surg. Obes. Relat. Dis. 3:2184–88 [Google Scholar]
  89. Walmsley PT, Sackett PR. 2013. Factors affecting potential personality retest improvement after initial failure. Hum. Perform. 26:5390–408 [Google Scholar]
  90. Wernimont PF, Campbell JP. 1968. Signs, samples, and criteria. J. Appl. Psychol. 52:372–76 [Google Scholar]
  91. Windle C. 1954. Test-retest effect on personality questionnaires. Educ. Psychol. Meas. 14:617–33 [Google Scholar]
  92. Wolkowitz AA. 2011. Multiple attempts on a nursing admissions examination: effects on the total score. J. Nurs. Educ. 50:9493–501 [Google Scholar]
/content/journals/10.1146/annurev-orgpsych-032516-113349
Loading
Retaking Employment Tests: What We Know and What We Still Need to Know
Annual Review of Organizational Psychology and Organizational Behavior 4, 445 (2017); https://doi.org/10.1146/annurev-orgpsych-032516-113349
/content/journals/10.1146/annurev-orgpsych-032516-113349
/content/journals/10.1146/annurev-orgpsych-032516-113349
Loading

Data & Media loading...

  • Article Type: Review Article

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error