1932
0
  1. Home
  2. A-Z Publications
  3. Annual Review of Organizational Psychology and Organizational Behavior
  4. Volume 2, 2015
  5. Article

Annual Review of Organizational Psychology and Organizational Behavior

Volume 2, 2015

Low-Fidelity Simulations

Abstract

Low-fidelity simulations, which combine closed-ended response options with realistic depictions of key job tasks, have grown in popularity for several practical reasons. Research into low-fidelity simulations, which has focused primarily on text-based situational judgment tests (SJTs), has shown that such measures () can predict a wide range of criteria, doing so with moderate adverse impact, depending on the construct saturation of the items, () are no more susceptible to coaching/cheating/retesting effects than many other forms of assessment and far less susceptible than some, and () continue to struggle when used to measure a single construct. Practice, by contrast, has moved well beyond text-based SJTs to include multimedia SJTs, online assessment center exercises, and game-like assessments. Because these are perceived more favorably by applicants, the same research dedicated to text-based SJTs is needed to guide development of these newer forms of low-fidelity simulation.

Keyword(s): low-fidelity simulations, situational judgment tests, SJTs
Loading

Article metrics loading...

/content/journals/10.1146/annurev-orgpsych-032414-111304
2015-04-10
2024-04-25
Loading full text...

Full text loading...

/deliver/fulltext/orgpsych/2/1/annurev-orgpsych-032414-111304.html?itemId=/content/journals/10.1146/annurev-orgpsych-032414-111304&mimeType=html&fmt=ahah

Literature Cited

  1. Anderson N, Salgado JF, Hülsheger UR. 2010. Applicant reactions in selection: comprehensive meta‐analysis into reaction generalization versus situational specificity. Int. J. Sel. Assess. 18:3291–304 [Google Scholar]
  2. Arthur W Jr, Doverspike D, Muñoz G, Taylor JE, Carr AE. 2014a. The use of mobile devices in high-stakes remotely delivered assessments and testing. Int. J. Sel. Assess. 22:2113–23 [Google Scholar]
  3. Arthur W Jr, Glaze RM, Jarrett SM, White CD, Schurig I, Taylor JE. 2014b. Comparative evaluation of three situational judgment test response formats in terms of construct-related validity, subgroup differences, and susceptibility to response distortion. J. Appl. Psychol 99:3535–45 [Google Scholar]
  4. Arthur W Jr, Villado AJ. 2008. The importance of distinguishing between constructs and methods when comparing predictors in personnel selection research and practice. J. Appl. Psychol 93:435–42 [Google Scholar]
  5. Bass BM, Avolio BJ. 1994. Improving Organizational Effectiveness Through Transformational Leadership Thousand Oaks, CA: Sage
  6. Bass BM, Avolio BJ. 1995. MLQ Multifactor Leadership Questionnaire: Technical Report Redwood City, CA: Mind Garden
  7. Becker TE. 2005. Development and validation of a situational judgment test of employee integrity. Int. J. Sel. Assess. 13:3225–32 [Google Scholar]
  8. Bergman ME, Drasgow F, Donovan MA, Henning JB, Juraska SE. 2006. Scoring situational judgment tests: Once you get the data, your troubles begin. Int. J. Sel. Assess. 14:223–35 [Google Scholar]
  9. Bledlow R, Frese M. 2009. A situational judgment test of personal initiative and its relationship to performance. Pers. Psychol 62:229–58 [Google Scholar]
  10. Burstein J, Chodorow M. 2010. Progress and new directions in technology for automated essay evaluation. The Oxford Handbook of Applied Linguistics Kaplan RB. 487–97 New York: Oxford Univ. Press, 2nd ed. [Google Scholar]
  11. Callinan M, Robertson IT. 2000. Work sample testing. Int. J. Sel. Assess. 8:248–60 [Google Scholar]
  12. Campion MC, Ployhart RE, MacKenzie WI Jr. 2014. The state of research on situational judgment tests: a content analysis and directions for future research. Hum. Perform 27:4283–310 [Google Scholar]
  13. Catano VM, Brochu A, Lamerson CD. 2012. Assessing the reliability of situational judgment tests used in high-stakes situations. Int. J. Sel. Assess 20:333–46 [Google Scholar]
  14. Chan D. 2006. Interactive effects of situational judgment effectiveness and proactive personality on work perceptions and work outcomes. J. Appl. Psychol. 91:2475–81 [Google Scholar]
  15. Chan D, Schmitt N. 1997. Video-based versus paper-and-pencil method of assessment in situational judgment tests: subgroup differences in test performance and face validity perceptions. J. Appl. Psychol. 82:1143–59 [Google Scholar]
  16. Christian MS, Edwards BD, Bradley JC. 2010. Situational judgment tests: constructs assessed and a meta-analysis of their criterion related validities. Pers. Psychol. 63:83–117 [Google Scholar]
  17. de Meijer L, Born M, Van Zielst J, Van der Molen HT. 2010. Construct-driven development of a video-based situational judgment test for integrity: a study in a multi-ethnic police setting. Eur. Psychol 15:229–36 [Google Scholar]
  18. Dunlop PD, Morrison DL, Cordery JL. 2011. Investigating retesting effects in a personnel selection context. Int. J. Sel. Assess. 19:2217–21 [Google Scholar]
  19. Fox J. 2010. What does a corporate web video cost? 25 factors (with prices) that affect corporate video production costs. OMM Blog, Mar. 3. http://onemarketmedia.com/blog/2010/03/what-does-a-web-video-cost-25-factors-with-prices-that-affect-video-production-costs/
  20. García PE, Olea J, De la Torre J. 2014. Application of cognitive diagnosis models to competency-based situational judgment tests. Psicothema 26:372–77 [Google Scholar]
  21. Goldstein IL, Zedeck S, Schneider B. 1993. An exploration of the job analysis-content validity process. Personnel Selection in Organizations Schmitt N, Borman WC. 7–10 San Francisco: Jossey-Bass [Google Scholar]
  22. Guenole N, Chernyshenko O, Stark S, Drasgow F. 2014. Are predictions based on situational judgement tests precise enough for feedback in leadership development?. Eur. J. Work Organ. Psychol. doi: 10.1080/1359432X.2014.926890 [Google Scholar]
  23. Hawkes B. 2012. Multimedia situational judgment tests: Are animation and live action really equivalent? Presented at Annu. Conf. Soc. Ind. Organ. Psychol., 26th, Apr. 26–28, San Diego
  24. Hawkes B. 2013. Test-takers’ empathy for animated humans in SJTs. Presented at Annu. Conf. Soc. Ind. Organ. Psychol., 27th, Apr. 11–13, Houston
  25. IDC 2013. Tablet shipments forecast to top total PC shipments in the fourth quarter of 2013 and annually by 2015, according to IDC. IDC Press Release, Sep. 11. http://www.idc.com/getdoc.jsp?containerId=prUS24314413
  26. Jones C, DeCotiis T. 1986. Video-assisted selection of hospitality employees. Cornell Hotel Restaur. Adm. Q. 27:267–73 [Google Scholar]
  27. King DD, Ryan AM, Kantrowitz T, Grelle D. 2014. MIT versus PCIT: assessing equivalence, individual differences, and reactions. Presented at Annu. Conf. Soc. Ind. Organ. Psychol., 29th, May 15–17, Honolulu, HI
  28. Kock N. 2005. Media richness or media naturalness? The evolution of our biological communication apparatus and its influence on our behavior toward e-communication tools. IEEE Trans. Prof. Commun. 48:2117–30 [Google Scholar]
  29. Laumer S, Eckhardt A, Weitzel T. 2012. Online gaming to find a new job—examining job seekers’ intention to use serious games as a self-assessment tool. Ger. J. Res. Hum. Resour. Manag. 26:3218–40 [Google Scholar]
  30. Less Rain 2012. Insanely Driven. Interact. Film, accessed June 24, 2014. http://www.lessrain.co.uk/#/work/insanely-driven
  31. Lievens F, Anseel F. 2007. Creating alternate in-basket forms through cloning: some preliminary results. Int. J. Sel. Assess. 15:4428–33 [Google Scholar]
  32. Lievens F, Buyse T, Sackett PR. 2005. The operational validity of a video-based situational judgment test for medical college admissions: illustrating the importance of matching predictor and criterion construct domains. J. Appl. Psychol. 90:442–52 [Google Scholar]
  33. Lievens F, Buyse T, Sackett PR, Connelly BS. 2012a. The effects of coaching on situational judgment tests in high-stakes selection. Int. J. Sel. Assess 20:272–82 [Google Scholar]
  34. Lievens F, De Corte W, Westerveld L. 2012b. Understanding the building blocks of selection procedures effects of response fidelity on performance and validity. J. Manag In press. doi: 10.1177/0149206312463941 [Google Scholar]
  35. Lievens F, Patterson F. 2011. The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection. J. Appl. Psychol. 96:927–40 [Google Scholar]
  36. Lievens F, Peeters H. 2008. Interviewers’ sensitivity to impression management tactics in structured interviews. Eur. J. Psychol. Assess 24:174–180 [Google Scholar]
  37. Lievens F, Sackett PR. 2006. Video-based versus written situational judgment tests: a comparison in terms of predictive validity. J. Appl. Psychol. 91:1181–88 [Google Scholar]
  38. Lievens F, Sackett PR. 2007. Situational judgment tests in high-stakes settings: issues and strategies with generating alternate forms. J. Appl. Psychol. 9:1043–55 [Google Scholar]
  39. Lievens F, Sackett PR. 2012. The validity of interpersonal skills assessment via situational judgment tests for predicting academic success and job performance. J. Appl. Psychol. 97:460–68 [Google Scholar]
  40. Lievens F, Van Keer E, Volckaert E. 2010. Gathering behavioral samples through a computerized and standardized assessment center exercise. J. Pers. Psychol. 9:294–98 [Google Scholar]
  41. Little Sister Films 2014. Into Deloitte. Interact. Video, accessed June 24, 2014. http://www.littlesisterfilms.co.nz/case-studies/2014/6/into-deloitte-interactive-video
  42. MacDorman KF, Coram JA, Ho C, Patel H. 2010. Gender differences in the impact of presentational factors in human character animation on decisions in ethical dilemmas. Presence 19:3213–29 [Google Scholar]
  43. MacKenzie WI Jr, Ployhart RE, Weekley JA, Ehlers C. 2010. Contextual effects on SJT responses: an examination of construct validity and mean differences across applicant and incumbent contexts. Hum. Perform. 23:11–21 [Google Scholar]
  44. Marentette BJ, Meyers LS, Hurtz GM, Kuang DC. 2012. Order effects on situational judgment test items: a case of construct—irrelevant difficulty. Int. J. Sel. Assess. 20:319–32 [Google Scholar]
  45. McDaniel MA, Hartman NS, Whetzel DL, Grubb W. 2007. Situational judgment tests, response instructions, and validity: a meta-analysis. Pers. Psychol. 60:63–91 [Google Scholar]
  46. McDaniel MA, Morgeson FP, Finnegan EB, Campion MA, Braverman EP. 2001. Use of situational judgment tests to predict job performance: a clarification of the literature. J. Appl. Psychol. 86:730–40 [Google Scholar]
  47. McDaniel MA, Psotka J, Legree PJ, Yost AP, Weekley JA. 2011. Toward an understanding of situational judgment item validity and group differences. J. Appl. Psychol. 96:327–36 [Google Scholar]
  48. Morgeson FP, Reider MH, Campion MA. 2005. Selecting individuals in team settings: the importance of social skills, personality characteristics, and teamwork knowledge. Pers. Psychol. 58:583–611 [Google Scholar]
  49. Motowidlo SJ, Beier ME. 2010. Differentiating specific job knowledge from implicit trait policies in procedural knowledge measured by a situational judgment test. J. Appl. Psychol. 95:2321–33 [Google Scholar]
  50. Motowidlo SJ, Crook AE, Kell HJ, Naemi B. 2009. Measuring procedural knowledge more simply with a single-response situational judgment test. J. Bus. Psychol. 24:281–88 [Google Scholar]
  51. Motowidlo SJ, Hooper AC, Jackson HL. 2006. Implicit policies about relations between personality traits and behavioral effectiveness in situational judgment items. J. Appl. Psychol. 91:4749–61 [Google Scholar]
  52. Mumford TV, Van Iddekinge CH, Morgeson FP, Campion MA. 2008. The team role test: development and validation of a team role knowledge situational judgment test. J. Appl. Psychol. 93:250–67 [Google Scholar]
  53. Nguyen NT, Biderman MD, McDaniel MA. 2005. Effects of response instructions on faking a situational judgment test. Int. J. Sel. Assess. 13:4250–60 [Google Scholar]
  54. Oostrom J, Born M, Serlie A, Van der Molen H. 2011. A multimedia situational test with a constructed-response format: Its relationship with personality, cognitive ability, job experience, and academic performance. J. Pers. Psychol. 10:278–88 [Google Scholar]
  55. Parshall C, Harmes JC. 2009. Improving the quality of innovative item types: four tasks for design and development. J. Appl. Test. Technol. 10:11–20 [Google Scholar]
  56. Peus C, Braun S, Frey D. 2013. Situation-based measurement of the full range of leadership model—development and validation of a situational judgment test. Leadersh. Q. 24:777–95 [Google Scholar]
  57. Ployhart RE, Holtz BC. 2008. The diversity–validity dilemma: strategies for reducing racioethnic and sex subgroup differences and adverse impact in selection. Pers. Psychol. 61:153–72 [Google Scholar]
  58. Ployhart RE, MacKenzie WI. 2011. Situational judgment tests: a critical review and agenda for the future. APA Handbook of Industrial and Organizational Psychology, Vol. 2: Selecting and Developing Members of the Organization Zedeck S. 237–52 Washington, DC: APA Books [Google Scholar]
  59. Roth PL, Bobko P, Buster MA. 2013. Situational judgment tests: the influence and importance of applicant status and targeted constructs on estimates of Black-White subgroup differences. J. Occup. Organ. Psychol 86:394–409 [Google Scholar]
  60. Sackett P, Lievens F. 2008. Personnel selection. Annu. Rev. Psychol. 59:419–50 [Google Scholar]
  61. Schmidt FL, Kaplan LB. 1971. Composite vs. multiple criteria: a review and resolution of the controversy. Pers. Psychol 24:419–34 [Google Scholar]
  62. Schmitt N, Chan D. 2006. Situational judgment tests: method or construct? See Weekley & Ployhart 2006, pp. 135–55
  63. Sharma S, Gangopadhyay M, Austin E, Mandal MK. 2013. Development and validation of a situational judgment test of emotional intelligence. Int. J. Sel. Assess. 21:57–73 [Google Scholar]
  64. Slaughter JE, Christian MS, Podsakoff NP, Sinar EF, Lievens F. 2014. On the limitations of using situational judgment tests to measure interpersonal skills: the moderating influence of employee anger. Pers. Psychol. 67:4847–85 [Google Scholar]
  65. StatCounter 2014. StatCounter Global Stats: comparison from June 2013 to May 2014. Accessed June 24, 2014. http://gs.statcounter.com/#desktop+mobile-comparison-ww-monthly-201306-201405
  66. Stevens MJ, Campion MA. 1999. Staffing work teams: development and validation of a selection test for teamwork. J. Manag. 25:2207–28 [Google Scholar]
  67. Thiessen-Roe A. 2013. Modeling situational judgment items with multiple distractor dimensions. New Developments in Quantitative Psychology Millsap RE, van der Ark LA, Bolt DM, Woods CM. 251–65 New York: Springer Sci. + Bus. Media [Google Scholar]
  68. Thornton G, Rupp D. 2006. Assessment Centers in Human Resource Management Mahwah, NJ: Erlbaum
  69. TMP Worldwide 2010. Reveal by L’Oréal. Case Study, accessed Apr. 24, 2014. http://www.tmp.com/upload/library/2780_L'Oreal_Reveal_Case-Study_2010-04-07_APPROVED.pdf
  70. Van Lankveld G, Schreurs S, Spronck PHM, Van den Herik H. 2011. Extraversion in games. Computers and Games (CG 2011) Van den Herik H, Lida H, Plaat A. 263–75 (LNCS Vol. 6515). Berlin: Springer [Google Scholar]
  71. Weekley JA, Jones C. 1997. Video-based situational testing. Pers. Psychol. 50:125–49 [Google Scholar]
  72. Weekley JA, Jones C. 1999. Further studies of situational testing. Pers. Psychol 52:679–700 [Google Scholar]
  73. Weekley JA, Ployhart RE. 2005. Situational judgment: antecedents and relationships with performance. Hum. Perform. 18:181–104 [Google Scholar]
  74. Weekley JA, Ployhart RE. 2006. Situational Judgment Tests: Theory, Measurement, and Application Mahwah, NJ: Erlbaum
  75. Weekley JA, Ployhart RE, Harold CM. 2004. Personality and situational judgment tests across applicant and incumbent settings: an examination of validity, measurement, and subgroup differences. Hum. Perform. 17:4433–61 [Google Scholar]
  76. Weekley JA, Ployhart RE, Holtz BC. 2006. On the development of situational judgment tests: issues in item development, scaling, and scoring. See Weekley & Ployhart 2006, pp. 157–82
  77. Westring AJF, Oswald FL, Schmitt N, Drzakowski S, Imus A et al. 2009. Estimating trait and situational variance in a situational judgment test. Hum. Perform. 22:44–63 [Google Scholar]
  78. Whetzel DL, McDaniel MA, Nguyen NT. 2008. Subgroup differences in situational judgment test performance: A meta-analysis. Hum. Perform. 21:291–309 [Google Scholar]
  79. Yee N. 2014. The Proteus Paradox New Haven, CT: Yale Univ. Press
  80. Zanbaka C, Ulinski A, Goolkasian P, Hodges LF. 2004. Effects of virtual human presence on task performance. Proc. Int. Conf. Artif. Real. Teleexistance (ICAT 2004), 14th, Seoul, Korea, pp. 174–81. Tokyo: Virtual Real. Soc. Japan
  81. Zu J, Kyllonen PC. 2012. Scoring situational judgment tests with item response models. Rep. ETS-2012-0160,R1, Educ. Testing Serv., Princeton, NJ
/content/journals/10.1146/annurev-orgpsych-032414-111304
Loading
Low-Fidelity Simulations
Annual Review of Organizational Psychology and Organizational Behavior 2, 295 (2015); https://doi.org/10.1146/annurev-orgpsych-032414-111304
/content/journals/10.1146/annurev-orgpsych-032414-111304
/content/journals/10.1146/annurev-orgpsych-032414-111304
Loading

Data & Media loading...

  • Article Type: Review Article

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error