1932

Abstract

Substantial mean score differences and significant adverse impact have long motivated the question of whether cognitive ability tests are biased against certain non-White subgroups. This article presents a framework for understanding the interrelated issues of adverse impact and test bias, with particular focus on two forms of test bias especially relevant for personnel selection: differential validity and differential prediction. Ethical and legal reasons that organizations should be concerned about differential validity/prediction are discussed. This article also serves as a critical review of the research literature on differential validity/prediction. The general conclusion is that available evidence supports the existence of differential validity/prediction in the form of correlation/slope and intercept differences between White and non-White subgroups. Implications for individuals and organizations are outlined, and a future research agenda is proposed highlighting the need for new, better data; new, better methods of testing for differential validity/prediction; and investigation of substantive factors causing differential validity/prediction.

Loading

Article metrics loading...

/content/journals/10.1146/annurev-orgpsych-032414-111256
2015-04-10
2024-06-23
Loading full text...

Full text loading...

/deliver/fulltext/orgpsych/2/1/annurev-orgpsych-032414-111256.html?itemId=/content/journals/10.1146/annurev-orgpsych-032414-111256&mimeType=html&fmt=ahah

Literature Cited

  1. AERA (Am. Educ. Res. Assoc.), APA (Am. Psychol. Assoc.), NCME (Natl. Counc. Meas. Educ.) 1999. Standards for Educational and Psychological Testing Washington, DC: AERA [Google Scholar]
  2. Aguinis H, Beaty JC, Boik RJ, Pierce CA. 2005. Effect size and power in assessing moderating effects of categorical variables using multiple regression: a 30-year review. J. Appl. Psychol. 90:94–107 [Google Scholar]
  3. Aguinis H, Culpepper SA, Pierce CA. 2010. Revival of test bias research in preemployment testing. J. Appl. Psychol. 95:648–80 [Google Scholar]
  4. Aguinis H, Smith MA. 2007. Understanding the impact of test validity and bias on selection errors and adverse impact in human resource selection. Pers. Psychol. 60:165–99 [Google Scholar]
  5. Aguinis H, Stone-Romero EF. 1997. Methodological artifacts in moderated multiple regression and their effects on statistical power. J. Appl. Psychol. 82:192–206 [Google Scholar]
  6. Bartlett CJ, Bobko P, Mosier SB, Hannan R. 1978. Testing for fairness with a moderated multiple regression strategy: an alternative to differential analysis. Pers. Psychol. 31:233–41 [Google Scholar]
  7. Berry CM. 2007. Toward an understanding of evidence of differential validity of cognitive ability tests for racial/ethnic subgroups. PhD Thesis, Univ. Minn., Minneap
  8. Berry CM, Barratt CL, Dovalina CL, Zhao P. 2014a. Can racial/ethnic subgroup criterion-to-test standard deviation ratios account for conflicting differential validity and differential prediction evidence for cognitive ability tests?. J. Occup. Organ. Psychol. 87:208–20 [Google Scholar]
  9. Berry CM, Clark MA, McClure TK. 2011. Racial/ethnic differences in the criterion-related validity of cognitive ability tests: a qualitative and quantitative review. J. Appl. Psychol. 96:881–906 [Google Scholar]
  10. Berry CM, Cullen MJ, Meyer JM. 2014b. Racial/ethnic subgroup differences in cognitive ability test range restriction: implications for differential validity. J. Appl. Psychol. 99:21–37 [Google Scholar]
  11. Berry CM, Kim A, Wang Y, Thompson R, Mobley WH. 2013a. Five-factor model personality measures and sex-based differential prediction of performance. Appl. Psychol. 62:13–43 [Google Scholar]
  12. Berry CM, Sackett PR, Sund A. 2013b. The role of range restriction and criterion contamination in assessing differential validity by race/ethnicity. J. Bus. Psychol. 28:345–59 [Google Scholar]
  13. Berry CM, Zhao P. 2015. Addressing criticisms of existing predictive bias research: Cognitive ability test scores still overpredict African Americans’ job performance. J. Appl. Psychol. 100:162–79 [Google Scholar]
  14. Bobko P, Roth PL, Potosky D. 1999. Derivation and implications of a meta-analytic matrix incorporating cognitive ability, alternative predictors, and job performance. Pers. Psychol. 52:561–89 [Google Scholar]
  15. Canivez GL, Konold TR. 2001. Assessing differential prediction bias in the Developing Cognitive Abilities Test across gender, race/ethnicity, and socioeconomic groups. Educ. Psychol. Meas. 61:159–71 [Google Scholar]
  16. Chen GW, Casper WJ, Cortina JM. 2001. The roles of self-efficacy and task complexity in relationships among cognitive ability, conscientiousness, and work-related performance: a meta-analytic examination. Hum. Perform. 14:3209–30 [Google Scholar]
  17. Gatewood RD, Field HS, Barrick MR. 2008. Human Resource Selection Mason, OH: South-West, 6th ed.. [Google Scholar]
  18. Hartigan JA, Wigdor AK. 1989. Differential validity and differential prediction. Fairness in Employment Testing: Validity Generalization, Non-White Issues, and the General Aptitude Test Battery Hartigan JA, Wigdor AK. 172–88 Washington, DC: Natl. Acad. Press [Google Scholar]
  19. Hough LM, Oswald FL, Ployhart RE. 2001. Determinants, detection and amelioration of adverse impact in personnel selection procedures: issues, evidence and lessons learned. Int. J. Sel. Assess. 9:152–94 [Google Scholar]
  20. Huffcutt AI, Roth PL. 1998. Racial group differences in employment interview evaluations. J. Appl. Psychol. 83:179–89 [Google Scholar]
  21. Hunter JE, Schmidt FL. 1976. Critical analysis of the statistical and ethical implications of various definitions of test bias. Psychol. Bull. 83:1053–71 [Google Scholar]
  22. Hunter JE, Schmidt FL. 1978. Differential and single-group validity of employment tests by race: a critical analysis of three recent studies. J. Appl. Psychol. 63:1–11 [Google Scholar]
  23. Hunter JE, Schmidt FL. 2004. Methods of Meta-Analysis: Correcting Error and Bias in Research Findings New York: Sage, 2nd ed.. [Google Scholar]
  24. Hunter JE, Schmidt FL, Hunter R. 1979. Differential validity of employment tests by race: a comprehensive review and analysis. Psychol. Bull. 86:721–35 [Google Scholar]
  25. Hunter JE, Schmidt FL, Le H. 2006. Implications of direct and indirect range restriction for meta-analysis methods and findings. J. Appl. Psychol. 91:594–612 [Google Scholar]
  26. Katzell RA, Dyer FJ. 1977. Differential validity revived. J. Appl. Psychol. 62:137–45 [Google Scholar]
  27. Kepes S, Banks GC, McDaniel M, Whetzel DL. 2012. Publication bias in the organizational sciences. Organ. Res. Methods 15:624–62 [Google Scholar]
  28. Kim A, Berry CM. 2015. Individual differences in social dominance orientation predict support for the use of cognitive ability tests. J. Personal. 83:14–25 [Google Scholar]
  29. Lautenschlager GJ, Mendoza JL. 1986. A step-down hierarchical multiple regression analysis for examining hypotheses about test bias in prediction. Appl. Psychol. Meas. 10:133–39 [Google Scholar]
  30. Linn RL. 1983. Pearson selection formulas: implications for studies of predictive bias and estimates of educational effects in selected samples. J. Educ. Meas. 20:1–15 [Google Scholar]
  31. Mattern KD, Patterson BF. 2013. Test of slope and intercept bias in college admissions: a response to Aguinis, Culpepper, and Pierce (2010). J. Appl. Psychol. 98:134–47 [Google Scholar]
  32. McDaniel MA, Kepes S, Banks GC. 2011. The Uniform Guidelines are a detriment to the field of personnel selection. Ind. Organ. Psychol. 4:494–514 [Google Scholar]
  33. McKay PF, McDaniel MA. 2006. A reexamination of Black-White mean differences in work performance: more data, more moderators. J. Appl. Psychol. 91:538–54 [Google Scholar]
  34. Murphy KR, Jacobs RR. 2012. Using effect size measures to reform the determination of adverse impact in equal employment litigation. Psychol. Public Policy Law 18:477–99 [Google Scholar]
  35. O’Connor EJ, Wexley KN, Alexander RA. 1975. Single-group validity: fact or fallacy?. J. Appl. Psychol. 60:352–55 [Google Scholar]
  36. O’Neill KA, McPeek WM. 1993. Item and test characteristics that are associated with differential item functioning. Differential Item Functioning Holland PW, Wainer H. 255–76 Hillsdale, NJ: Erlbaum [Google Scholar]
  37. Patterson BF, Mattern KD. 2011. Validity of the SAT for predicting first-year grades: 2008 SAT validity sample. Stat. Rep. No. 2011–5, Coll. Board, New York
  38. Ployhart RE, Holtz BC. 2008. The diversity-validity dilemma: strategies for reducing racioethnic and sex subgroup differences and adverse impact in selection. Pers. Psychol. 61:153–71 [Google Scholar]
  39. Roth PL, Bevier CA, Bobko P, Switzer FS, Tyler P. 2001. Ethnic group differences in cognitive ability in employment and educational settings: a meta-analysis. Pers. Psychol. 54:297–330 [Google Scholar]
  40. Roth PL, Huffcutt AI, Bobko P. 2003. Ethnic group differences in measures of job performance: a new meta-analysis. J. Appl. Psychol. 88:694–706 [Google Scholar]
  41. Roth PL, Le H, Oh I, Van Iddekinge CH, Buster MA et al. 2014. Differential validity for cognitive ability tests in employment and educational settings: not much more than range restriction?. J. Appl. Psychol. 99:1–20 [Google Scholar]
  42. Rotundo M, Sackett PR. 1999. Effect of rater race on conclusions regarding differential prediction in cognitive ability tests. J. Appl. Psychol. 84:815–22 [Google Scholar]
  43. Saad S, Sackett PR. 2002. Investigating differential prediction by gender in employment-oriented personality measures. J. Appl. Psychol. 87:667–74 [Google Scholar]
  44. Sackett PR, Borneman MJ, Connelly BS. 2008. High stakes testing in higher education and employment: appraising the evidence for validity and fairness. Am. Psychol. 63:215–27 [Google Scholar]
  45. Sackett PR, Schmitt N, Ellingson JE, Kabin MB. 2001. High-stakes testing in employment, credentialing, and higher education: prospects in a post-affirmative-action world. Am. Psychol. 56:302–18 [Google Scholar]
  46. Schmidt FL, Hunter JE. 1981. Employment testing: old theories and new research findings. Am. Psychol. 36:1128–37 [Google Scholar]
  47. Schmidt FL, Hunter JE. 1998. The validity and utility of selection methods in personnel psychology: practical and theoretical implications of 85 years of research findings. Psychol. Bull. 124:262–74 [Google Scholar]
  48. Schmidt FL, Pearlman K, Hunter JE. 1980. The validity and fairness of employment and educational tests for Hispanic Americans: a review and analysis. Pers. Psychol. 33:705–24 [Google Scholar]
  49. Schmitt N. 2014. Personality and cognitive ability as predictors of effective performance at work. Annu. Rev. Organ. Psychol. Organ. Behav. 1:45–65 [Google Scholar]
  50. Schult J, Hell B, Passler K, Schuler H. 2013. Sex-specific differential prediction of academic achievement by German ability tests. Int. J. Sel. Assess. 21:130–34 [Google Scholar]
  51. Sidanius J, Pratto F. 1999. Social Dominance: An Intergroup Theory of Social Hierarchy and Oppression New York: Cambridge Univ. Press [Google Scholar]
  52. SIOP (Soc. Ind. Organ. Psychol.) 2003. Principles for the Validation and Use of Personnel Selection Procedures Bowling Green, OH: SIOP, 4th ed. [Google Scholar]
  53. Steele CM, Aronson J. 1995. Stereotype threat and the intellectual test performance of African Americans. J. Personal. Soc. Psychol. 69:797–811 [Google Scholar]
  54. Whetzel DL, McDaniel MA, Nguyen NT. 2008. Subgroup differences in situational judgment test performance: a meta-analysis. Hum. Perform. 21:291–309 [Google Scholar]
/content/journals/10.1146/annurev-orgpsych-032414-111256
Loading
/content/journals/10.1146/annurev-orgpsych-032414-111256
Loading

Data & Media loading...

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error