1932
0
  1. Home
  2. A-Z Publications
  3. Annual Review of Organizational Psychology and Organizational Behavior
  4. Volume 9, 2022
  5. Article

Annual Review of Organizational Psychology and Organizational Behavior

Volume 9, 2022

Accumulating Knowledge in the Organizational Sciences

Abstract

In some fields, research findings are rigorously curated in a common language and made available to enable future use and large-scale, robust insights. Organizational researchers have begun such efforts [e.g., metaBUS ()] but are far from the efficient, comprehensive curation seen in areas such as cognitive neuroscience or genetics. This review provides a sample of insights from research curation efforts in organizational research, psychology, and beyond—insights not possible by even large-scale, substantive meta-analyses. Efforts are classified as either science-of-science research or large-scale, substantive research. The various methods used for information extraction (e.g., from PDF files) and classification (e.g., using consensus ontologies) is reviewed. The review concludes with a series of recommendations for developing and leveraging the available corpus of organizational research to speed scientific progress.

Keyword(s): big data, meta-analysis, meta-science, research curation
Loading

Article metrics loading...

/content/journals/10.1146/annurev-orgpsych-012420-090657
2022-01-21
2024-04-18
Loading full text...

Full text loading...

/deliver/fulltext/orgpsych/9/1/annurev-orgpsych-012420-090657.html?itemId=/content/journals/10.1146/annurev-orgpsych-012420-090657&mimeType=html&fmt=ahah

Literature Cited

  1. Aguinis H, Beaty JC, Boik RJ, Pierce CA. 2005. Effect size and power in assessing moderating effects of categorical variables using multiple regression: a 30-year review. J. Appl. Psychol. 90:194–107
     [Google Scholar]
  2. Aguinis H, Dalton DR, Bosco FA, Pierce CA, Dalton CM 2011. Meta-analytic choices and judgment calls: implications for theory building and testing, obtained effect sizes, and scholarly impact. . J. Manag. 37:15–38
     [Google Scholar]
  3. Anseel F, Lievens F, Schollaert E, Choragwicka B. 2010. Response rates in organizational science, 1995–2008: a meta-analytic review and guidelines for survey researchers. J. Bus. Psychol. 25:3335–49
     [Google Scholar]
  4. Appelbaum M, Cooper H, Kline RB, Mayo-Wilson E, Nezu AM, Rao SM. 2018. Journal article reporting standards for quantitative research in psychology: the APA Publications and Communications Board task force report. Am. Psychol. 73:13–25
     [Google Scholar]
  5. Baum J, Bromiley P. 2019. P-hacking in top-tier management journals. Acad. Manag. Proc 2019:10810
     [Google Scholar]
  6. Bell G, Hey T, Szalay A. 2009. Beyond the data deluge. Science 323:59191297–98
     [Google Scholar]
  7. Bergmann C, Tsuji S, Cristia A. 2017. Top-down versus bottom-up theories of phonological acquisition: a big data approach. Proc. Interspeech 2017:2013–16
     [Google Scholar]
  8. Blanch A, García R, Planes J, Gil R, Balada F et al. 2017. Ontologies about human behavior: a review of knowledge modeling systems. Eur. Psychol. 22:3180–97
     [Google Scholar]
  9. Bosco FA, Aguinis H, Field JG, Pierce CA, Dalton DR 2016. HARKing's threat to organizational research: evidence from primary and meta-analytic sources. Pers. Psychol. 69:709–50
     [Google Scholar]
  10. Bosco FA, Aguinis H, Kepes S, Gabriel AS, Field JG. 2014. Assessing the impact of nonresponse bias: a “big science” approach Paper presented at the 74th Annual Meeting of the Academy of Management Philadelphia, PA: Aug. 1–5
  11. Bosco FA, Aguinis H, Singh K, Field JG, Pierce CA. 2015a. Correlational effect size benchmarks. J. Appl. Psychol. 100:431–49
     [Google Scholar]
  12. Bosco FA, Field JG, Larsen K, Chang Y, Uggerslev KL 2020. Advancing meta-analysis with knowledge management platforms: using metaBUS in psychology. Adv. Methods Pract. Psychol. Sci. 3:1124–37
     [Google Scholar]
  13. Bosco FA, Landis R, Kepes S, Uggerslev KL, Steel P, Brooks P. 2017. Dimension reduction as a vehicle for assessing construct redundancy Paper presented at the 32nd Annual Meeting of the Society for Industrial and Organizational Psychology Orlando, FL: Apr. 26–29
  14. Bosco FA, Steel P, Oswald FL, Uggerslev KL, Field JG. 2015b. Cloud-based meta-analysis to bridge science and practice: Welcome to metaBUS. Pers. Assess. Decis. 1:3–17
     [Google Scholar]
  15. Burgard T, Bošnjak M, Studtrucker R. 2021. Community-augmented meta-analyses (CAMAs) in Psychology. Zeitschrift Psychol 229:115–23
     [Google Scholar]
  16. Campbell Collaboration 2021. Evidence and gap maps (EGMs). Campbell Collaboration. https://www.campbellcollaboration.org/evidence-gap-maps.html
    [Google Scholar]
  17. Cariaso M, Lennon G. 2012. SNPedia: a wiki supporting personal genome annotation, interpretation and analysis. Nucleic Acids Res. 40:D1D1308–312
     [Google Scholar]
  18. Chamberlin M, Newton DW, Lepine JA. 2017. A meta-analysis of voice and its promotive and prohibitive forms: identification of key associations, distinctions, and future research directions. Pers. Psychol. 70:111–71
     [Google Scholar]
  19. Cohen J. 1962. The statistical power of abnormal-social psychological research: a review. J. Abnorm. Soc. Psychol. 65:3145–53
     [Google Scholar]
  20. Cohen J. 1988. Statistical Power Analysis for the Behavioral Sciences Hillsdale, NJ: Erlbaum:. , 2nd ed..
  21. Côté IM, Curtis PS, Rothstein HR, Stewart GB 2013. Gathering data: searching literature and selection criteria. Handbook of Meta-Analysis in Ecology and Evolution J Koricheva, J Gurevitch, K Mengersen 37–51 Princeton, NJ: Princeton Univ. Press
     [Google Scholar]
  22. Cox J, Oh EY, Simmons B, Lintott C, Masters K et al. 2015. Defining and measuring success in online citizen science: a case study of Zooniverse projects. . Comput. Sci. Eng. 17:428–41
     [Google Scholar]
  23. Culpepper SA, Aguinis H. 2011. R is for revolution: a cutting-edge, free, open source statistical package. Organ. Res. Methods 14:4735–40
     [Google Scholar]
  24. Eisenberg IW, Bissett PG, Canning JR, Dallery J, Enkavi AZ et al. 2018. Applying novel technologies and methods to inform the ontology of self-regulation. Behav. Res. Ther. 101:46–57
     [Google Scholar]
  25. Eisenberg IW, Bissett PG, Enkavi AZ, Li J, MacKinnon DP, Marsch LA, Poldrack RA. 2019. Uncovering the structure of self-regulation through data-driven ontology discovery. Nat. Commun. 10:12319
     [Google Scholar]
  26. Fanelli D. 2009. How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLOS ONE 4:5e5738
     [Google Scholar]
  27. Field JG, Bosco FA, Kepes S. 2021a. How robust is our cumulative knowledge on turnover?. J. Bus. Psychol. 36:349–65
     [Google Scholar]
  28. Field JG, Bosco FA, Kraichy D, Uggerslev KL, Geiger MK 2021b. More alike than different? A comparison of variance explained by cross-cultural models. J. Int. Bus. Stud. https://doi.org/10.1057/s41267-021-00428-z. In press
    [Crossref] [Google Scholar]
  29. Field JG, Bosco FA, Pierce CA 2013. Variability in effect-size magnitude as a function of sample type. Paper presented at the Annual Meeting of the Academy of Management Lake Buena Vista, FL: Aug. 9–13
  30. Field JG, Mihm DC, O'Boyle EH, Bosco FA, Uggerslev KL, Steel P. 2015. An examination of the funding-finding relation in the field of management Paper presented at the 17th Annual Meeting of the Academy of Management: Opening Governance Vancouver, BC: Aug. 7–11
  31. Furnas GW, Landauer TK, Gomez LM, Dumais ST. 1987. The vocabulary problem in human-system communication. Commun. ACM 30:11964–71
     [Google Scholar]
  32. Gene Ontology Consortium 2015. Gene ontology consortium: going forward. Nucleic Acids Res. 43:D1049–1056
     [Google Scholar]
  33. Gordon ME, Slade LA, Schmitt N. 1986. The “science of the sophomore” revisited: from conjecture to empiricism. Acad. Manag. Rev. 11:191–207
     [Google Scholar]
  34. Götz M, O'Boyle EH, Gonzalez-Mulé E, Banks GC, Bollmann SS. 2021. The “Goldilocks Zone”: (Too) many confidence intervals in tests of mediation just exclude zero. Psychol. Bull. 147:195–114
     [Google Scholar]
  35. Greco LM, O'Boyle EH, Cockburn BS, Yuan Z. 2018. Meta-analysis of coefficient alpha: a reliability generalization study. J. Manag. Stud. 55:583–618
     [Google Scholar]
  36. Hambrick DC. 2007. The field of management's devotion to theory: Too much of a good thing?. Acad. Manag. J. 50:61346–52
     [Google Scholar]
  37. Hardwicke TE, Serghiou S, Janiaud P, Danchev V, Crüwell S, Goodman SN, Ioannidis JP. 2020. Calibrating the scientific ecosystem through meta-research. Annu. Rev. Stat. Appl. 7:11–37
     [Google Scholar]
  38. Hastings J, Frishkoff GA, Smith B, Jensen M, Poldrack RA et al. 2014. Interdisciplinary perspectives on the development, integration, and application of cognitive ontologies. Front. Neuroinformatics 8:62
     [Google Scholar]
  39. He Y, Wang Y, Payne SC 2019. How is safety climate formed? A meta-analysis of the antecedents of safety climate. Organ. Psychol. Rev. 9:2–3124–56
     [Google Scholar]
  40. Head ML, Holman L, Lanfear R, Kahn AT, Jennions MD. 2015. The extent and consequences of p-hacking in science. PLOS Biol 13:3e1002106
     [Google Scholar]
  41. Hofstede G. 1980. Culture's Consequences: International Differences in Work-Related Values Beverly Hills, CA: Sage
  42. House RJ, Hanges PJ, Javidan M, Dorfman PW, Gupta V. 2004. Culture, Leadership, and Organizations Thousand Oaks, CA: Sage
  43. Hutchison KA, Balota DA, Neely JH, Cortese MJ, Cohen-Shikora ER et al. 2013. The semantic priming project. Behav. Res. Methods 45:41099–1114
     [Google Scholar]
  44. Ioannidis JPA. 2014. Discussion: Why “An estimate of the science-wise false discovery rate and application to the top medical literature” is false. Biostatistics 15:128–36
     [Google Scholar]
  45. Ioannidis JPA. 2018. Meta-research: why research on research matters. PLOS Biol 16:3e2005468
     [Google Scholar]
  46. Ioannidis JPA. 2019. What have we (not) learnt from millions of scientific papers with P values?. Am. Stat. 73:20–25
     [Google Scholar]
  47. John LK, Loewenstein G, Prelec D. 2012. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 23:5524–32
     [Google Scholar]
  48. Judge TA, Thoresen CJ, Bono JE, Patton GK. 2001. The job satisfaction–job performance relationship: a qualitative and quantitative review. Psychol. Bull. 127:3376–407
     [Google Scholar]
  49. Kelley TL. 1927. Interpretation of Educational Measurements Chicago: World Book Co.
  50. Kerr NL. 1998. HARKing: hypothesizing after the results are known. Pers. Soc. Psychol. Rev. 2:3196–217
     [Google Scholar]
  51. Kleine AK, Rudolph CW, Zacher H. 2019. Thriving at work: a meta-analysis. J. Organ. Behav. 40:9–10973–99
     [Google Scholar]
  52. Köhler T, Cortina JM, Kurtessis JN, Gölz M. 2015. Are we correcting correctly? Interdependence of reliabilities in meta-analysis. Organ. Res. Methods 18:355–428
     [Google Scholar]
  53. Laird AR, Lancaster JL, Fox PT. 2005. BrainMap: the social evolution of a human brain mapping database. Neuroinformatics 3:65–78
     [Google Scholar]
  54. Larsen KR, Michie S, Hekler EB, Gibson B, Spruijt-Metz D et al. 2017. Behavior change interventions: the potential of ontologies for advancing science and practice. J. Behav. Med. 40:16–22
     [Google Scholar]
  55. Le H, Schmidt FL, Harter JK, Lauver KJ. 2010. The problem of empirical redundancy of constructs in organizational research: an empirical investigation. Organ. Behav. Hum. Decis. Process. 112:2112–25
     [Google Scholar]
  56. Leavitt K, Mitchell T, Peterson J 2010. Theory pruning: strategies for reducing our dense theoretical landscape. Organ. Res. Methods 13:4644–67
     [Google Scholar]
  57. LeBel EP, McCarthy RJ, Earp BD, Elson M, Vanpeamel W. 2018. A unified framework to quantify the credibility of scientific findings. Adv. Methods Pract. Psychol. Sci. 1:3389–402
     [Google Scholar]
  58. Lee CISG, Bosco FA, Steel P, Uggerslev KL. 2017. A metaBUS enabled meta-analysis of career satisfaction. Career Dev. Int. 22:5565–82
     [Google Scholar]
  59. Lee CISG, Felps W, Baruch Y 2014. Toward a taxonomy of career studies through bibliometric visualization. J. Vocat. Behav. 85:3339–51
     [Google Scholar]
  60. Lewis ML, Braginsky M, Tsuji S, Bergmann C, Piccinini PE et al. 2016. A quantitative synthesis of early language acquisition using meta-analysis. PsyArXiv. https://psyarxiv.com/htsjm/
  61. Linnaeus C. 1735. Systema naturae, sive regna tria naturae systematice proposita per classes, ordines, genera, & species Leiden: de Groot
  62. Mackey JD, McAllister CP, Ellen BP III, Carson JE. 2019. A meta-analysis of interpersonal and organizational workplace deviance research. J. Manag. 47:3597–622
     [Google Scholar]
  63. Marshall IJ, Wallace BC. 2019. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Syst. Rev. 8:1163
     [Google Scholar]
  64. Meyer RD. 2015. Taxonomy of Situations and their Measurement Oxford: Oxford Libr. Psychol https://www.oxfordhandbooks.com/view/10.1093/oxfordhb/9780199935291.001.0001/oxfordhb-9780199935291-e-22
  65. Mischel W. 2008. The toothbrush problem. APS Observer Dec. 1. https://www.psychologicalscience.org/observer/the-toothbrush-problem
    [Google Scholar]
  66. Morris SB, DeShon RP. 2002. Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychol. Methods 7:1105–125
     [Google Scholar]
  67. Morse PJ, Neel R, Todd E, Funder D 2015. Renovating situation taxonomies: exploring the construction and content of fundamental motive situation types. J. Pers. 83:4389–403
     [Google Scholar]
  68. Müller-Wille S, Charmantier I 2012. Natural history and information overload: the case of Linnaeus. Stud. Hist. Philos. Sci C. 43:14–15
     [Google Scholar]
  69. Nuijten MB, Hartgerink CHJ, van Assen MALM, Epskamp S, Wicherts JM. 2016. The prevalence of statistical reporting errors in psychology (1985–2013). Behav. Res. Methods 48:1205–26
     [Google Scholar]
  70. Nuijten MB, van Assen MALM, Hartgerink CHJ, Epskamp S, Wicherts JM 2017. The validity of the tool “statcheck” in discovering statistical reporting inconsistencies. PsyArXiv https://psyarxiv.com/tcxaj/
    [Google Scholar]
  71. Parrigon S, Woo SE, Tay L, Wang T. 2017. CAPTION-ing the situation: a lexically-derived taxonomy of psychological situation characteristics. J. Pers. Soc. Psychol. 112:4642–81
     [Google Scholar]
  72. Peterson NG, Mumford MD, Borman WC, Jeanneret PR, Fleishman EA et al. 2001. Understanding work using the Occupational Information Network (O*NET): implications for practice and research. Pers. Psychol. 54:2451–92
     [Google Scholar]
  73. Petrosino A, Boruch RF, Soydan H, Duggan L, Sanchez-Meca J. 2001. Meeting the challenges of evidence-based policy: the Campbell Collaboration. Ann. Am. Acad. Political Soc. Sci. 578:114–34
     [Google Scholar]
  74. Platt JR. 1964. Strong inference. Science 146:3642347–53
     [Google Scholar]
  75. Poldrack RA, Yarkoni T. 2016. From brain maps to cognitive ontologies: informatics and the search for mental structure. Annu. Rev. Psychol. 67:587–612
     [Google Scholar]
  76. Rauthmann JF, Gallardo-Pujol D, Guillaume EM, Todd E, Nave CS et al. 2014. The Situational Eight DIAMONDS: a taxonomy of major dimensions of situation characteristics. J. Pers. Soc. Psychol. 107:4677–718
     [Google Scholar]
  77. Ronen S, Shenkar O 1985. Clustering countries on attitudinal dimensions: a review and synthesis. Acad. Manag. Rev. 10:3435–54
     [Google Scholar]
  78. Rudolph CW, Kooij DT, Rauvola RS, Zacher H. 2018. Occupational future time perspective: a meta-analysis of antecedents and outcomes. J. Organ. Behav. 39:2229–48
     [Google Scholar]
  79. Saran A, White H, Kuper H. 2020. Evidence and gap map of studies assessing the effectiveness of interventions for people with disabilities in low-and middle-income countries. Campbell Syst. Rev. 16:1e1070
     [Google Scholar]
  80. Schmidt T. 2017. Statcheck does not work: all the numbers. Reply to Nuijten et al. 2017. PsyArXiv. https://psyarxiv.com/hr6qy/
  81. Schriger DL, Chehrazi AC, Merchant RM, Altman DG. 2011. Use of the internet by print medical journals in 2003 to 2009: a longitudinal observational study. Ann. Emerg. Med. 57:2153–60
     [Google Scholar]
  82. Shaffer JA, DeGeest D, Li A 2016. Tackling the problem of construct proliferation: a guide to assessing the discriminant validity of conceptually related constructs. Organ. Res. Methods 19:180–110
     [Google Scholar]
  83. Spadaro G, Tiddi I, Columbus S, Jin S, Teije AT, Balliet D 2020. The Cooperation Databank. PsyArXiv. https://doi.org/10.31234/osf.io/rveh3
    [Crossref]
  84. Steel P, Schmidt J, Bosco FA, Uggerslev KL. 2019. The effects of personality on job satisfaction and life satisfaction: a meta-analytic investigation accounting for bandwidth–fidelity and commensurability. Hum. Relat. 72:217–47
     [Google Scholar]
  85. Steel P, Taras V, Uggerslev KL, Bosco FA. 2018. The happy culture: a theoretical, meta-analytic and empirical review of the relationship between culture and wealth and subjective wellbeing. Pers. Soc. Psychol. Rev. 22:128–69
     [Google Scholar]
  86. Tackett JL, Brandes CM, Dworak EM, Shields AN. 2020. Bringing the (pre) registration revolution to graduate training. Can. Psychol./Psychol. Can. 61:299–309
     [Google Scholar]
  87. van Aert RCM, Wicherts JM, van Assen MALM. 2019. Publication bias examined in meta-analyses from psychology and medicine: a meta-meta-analysis. PLOS ONE 14:4e0215052
     [Google Scholar]
  88. Van Iddekinge CH, Arnold JD, Frieder RE, Roth PL 2019. A meta-analysis of the criterion-related validity of prehire work experience. Pers. Psychol. 72:4571–98
     [Google Scholar]
  89. Vanasse TJ, Fox PM, Barron DS, Robertson M, Eickhoff SB, Lancaster JL, Fox PT. 2018. BrainMap VBM: an environment for structural meta-analysis. Hum. Brain Mapp. 39:83308–25
     [Google Scholar]
  90. Veenhoven R. 2009. World Database of Happiness: tool for dealing with the ‘data-deluge. Psychol. Top. 18:221–46
     [Google Scholar]
  91. Venter JC, Remington K, Heidelberg JF, Halpern AL, Rusch D et al. 2004. Environmental genome shotgun sequencing of the Sargasso Sea. Science 304:566766–74
     [Google Scholar]
  92. Wiernik BM, Ones DS. 2018. Ethical employee behaviors in the consensus taxonomy of counterproductive work behaviors. Int. J. Sel. Assess. 26:136–48
     [Google Scholar]
  93. Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M et al. 2016. Comment: The FAIR Guiding Principles for scientific data management and stewardship. Nat. Sci. Data 3:160018
     [Google Scholar]
  94. Woo SE, O'Boyle EH, Spector PE. 2017. Best practices in developing, conducting, and evaluating inductive research. Hum. Resour. Manag. Rev. 27:2255–64
     [Google Scholar]
  95. Woznyj HM, Banks GC, Dunn AM, Berka G, Woehr D. 2020. Re-introducing cognitive complexity: a meta-analysis and agenda for future research. Hum. Perform. 33:11–33
     [Google Scholar]
  96. Yarkoni T, Poldrack RA, Nichols TE, Van Essen DC, Wager TD. 2011. Large-scale automated synthesis of human functional neuroimaging data. Nat. Methods 8:8665–70
     [Google Scholar]
/content/journals/10.1146/annurev-orgpsych-012420-090657
Loading
Accumulating Knowledge in the Organizational Sciences
Annual Review of Organizational Psychology and Organizational Behavior 9, 441 (2022); https://doi.org/10.1146/annurev-orgpsych-012420-090657
/content/journals/10.1146/annurev-orgpsych-012420-090657
/content/journals/10.1146/annurev-orgpsych-012420-090657
Loading

Data & Media loading...

  • Article Type: Review Article

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error