1932

Abstract

In some fields, research findings are rigorously curated in a common language and made available to enable future use and large-scale, robust insights. Organizational researchers have begun such efforts [e.g., metaBUS ()] but are far from the efficient, comprehensive curation seen in areas such as cognitive neuroscience or genetics. This review provides a sample of insights from research curation efforts in organizational research, psychology, and beyond—insights not possible by even large-scale, substantive meta-analyses. Efforts are classified as either science-of-science research or large-scale, substantive research. The various methods used for information extraction (e.g., from PDF files) and classification (e.g., using consensus ontologies) is reviewed. The review concludes with a series of recommendations for developing and leveraging the available corpus of organizational research to speed scientific progress.

Loading

Article metrics loading...

/content/journals/10.1146/annurev-orgpsych-012420-090657
2022-01-21
2024-04-17
Loading full text...

Full text loading...

/deliver/fulltext/orgpsych/9/1/annurev-orgpsych-012420-090657.html?itemId=/content/journals/10.1146/annurev-orgpsych-012420-090657&mimeType=html&fmt=ahah

Literature Cited

  1. Aguinis H, Beaty JC, Boik RJ, Pierce CA. 2005. Effect size and power in assessing moderating effects of categorical variables using multiple regression: a 30-year review. J. Appl. Psychol. 90:194–107
    [Google Scholar]
  2. Aguinis H, Dalton DR, Bosco FA, Pierce CA, Dalton CM 2011. Meta-analytic choices and judgment calls: implications for theory building and testing, obtained effect sizes, and scholarly impact. . J. Manag. 37:15–38
    [Google Scholar]
  3. Anseel F, Lievens F, Schollaert E, Choragwicka B. 2010. Response rates in organizational science, 1995–2008: a meta-analytic review and guidelines for survey researchers. J. Bus. Psychol. 25:3335–49
    [Google Scholar]
  4. Appelbaum M, Cooper H, Kline RB, Mayo-Wilson E, Nezu AM, Rao SM. 2018. Journal article reporting standards for quantitative research in psychology: the APA Publications and Communications Board task force report. Am. Psychol. 73:13–25
    [Google Scholar]
  5. Baum J, Bromiley P. 2019. P-hacking in top-tier management journals. Acad. Manag. Proc 2019:10810
    [Google Scholar]
  6. Bell G, Hey T, Szalay A. 2009. Beyond the data deluge. Science 323:59191297–98
    [Google Scholar]
  7. Bergmann C, Tsuji S, Cristia A. 2017. Top-down versus bottom-up theories of phonological acquisition: a big data approach. Proc. Interspeech 2017:2013–16
    [Google Scholar]
  8. Blanch A, García R, Planes J, Gil R, Balada F et al. 2017. Ontologies about human behavior: a review of knowledge modeling systems. Eur. Psychol. 22:3180–97
    [Google Scholar]
  9. Bosco FA, Aguinis H, Field JG, Pierce CA, Dalton DR 2016. HARKing's threat to organizational research: evidence from primary and meta-analytic sources. Pers. Psychol. 69:709–50
    [Google Scholar]
  10. Bosco FA, Aguinis H, Kepes S, Gabriel AS, Field JG. 2014. Assessing the impact of nonresponse bias: a “big science” approach Paper presented at the 74th Annual Meeting of the Academy of Management Philadelphia, PA: Aug. 1–5
  11. Bosco FA, Aguinis H, Singh K, Field JG, Pierce CA. 2015a. Correlational effect size benchmarks. J. Appl. Psychol. 100:431–49
    [Google Scholar]
  12. Bosco FA, Field JG, Larsen K, Chang Y, Uggerslev KL 2020. Advancing meta-analysis with knowledge management platforms: using metaBUS in psychology. Adv. Methods Pract. Psychol. Sci. 3:1124–37
    [Google Scholar]
  13. Bosco FA, Landis R, Kepes S, Uggerslev KL, Steel P, Brooks P. 2017. Dimension reduction as a vehicle for assessing construct redundancy Paper presented at the 32nd Annual Meeting of the Society for Industrial and Organizational Psychology Orlando, FL: Apr. 26–29
  14. Bosco FA, Steel P, Oswald FL, Uggerslev KL, Field JG. 2015b. Cloud-based meta-analysis to bridge science and practice: Welcome to metaBUS. Pers. Assess. Decis. 1:3–17
    [Google Scholar]
  15. Burgard T, Bošnjak M, Studtrucker R. 2021. Community-augmented meta-analyses (CAMAs) in Psychology. Zeitschrift Psychol 229:115–23
    [Google Scholar]
  16. Campbell Collaboration 2021. Evidence and gap maps (EGMs). Campbell Collaboration. https://www.campbellcollaboration.org/evidence-gap-maps.html
    [Google Scholar]
  17. Cariaso M, Lennon G. 2012. SNPedia: a wiki supporting personal genome annotation, interpretation and analysis. Nucleic Acids Res. 40:D1D1308–312
    [Google Scholar]
  18. Chamberlin M, Newton DW, Lepine JA. 2017. A meta-analysis of voice and its promotive and prohibitive forms: identification of key associations, distinctions, and future research directions. Pers. Psychol. 70:111–71
    [Google Scholar]
  19. Cohen J. 1962. The statistical power of abnormal-social psychological research: a review. J. Abnorm. Soc. Psychol. 65:3145–53
    [Google Scholar]
  20. Cohen J. 1988. Statistical Power Analysis for the Behavioral Sciences Hillsdale, NJ: Erlbaum:. , 2nd ed..
  21. Côté IM, Curtis PS, Rothstein HR, Stewart GB 2013. Gathering data: searching literature and selection criteria. Handbook of Meta-Analysis in Ecology and Evolution J Koricheva, J Gurevitch, K Mengersen 37–51 Princeton, NJ: Princeton Univ. Press
    [Google Scholar]
  22. Cox J, Oh EY, Simmons B, Lintott C, Masters K et al. 2015. Defining and measuring success in online citizen science: a case study of Zooniverse projects. . Comput. Sci. Eng. 17:428–41
    [Google Scholar]
  23. Culpepper SA, Aguinis H. 2011. R is for revolution: a cutting-edge, free, open source statistical package. Organ. Res. Methods 14:4735–40
    [Google Scholar]
  24. Eisenberg IW, Bissett PG, Canning JR, Dallery J, Enkavi AZ et al. 2018. Applying novel technologies and methods to inform the ontology of self-regulation. Behav. Res. Ther. 101:46–57
    [Google Scholar]
  25. Eisenberg IW, Bissett PG, Enkavi AZ, Li J, MacKinnon DP, Marsch LA, Poldrack RA. 2019. Uncovering the structure of self-regulation through data-driven ontology discovery. Nat. Commun. 10:12319
    [Google Scholar]
  26. Fanelli D. 2009. How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLOS ONE 4:5e5738
    [Google Scholar]
  27. Field JG, Bosco FA, Kepes S. 2021a. How robust is our cumulative knowledge on turnover?. J. Bus. Psychol. 36:349–65
    [Google Scholar]
  28. Field JG, Bosco FA, Kraichy D, Uggerslev KL, Geiger MK 2021b. More alike than different? A comparison of variance explained by cross-cultural models. J. Int. Bus. Stud. https://doi.org/10.1057/s41267-021-00428-z. In press
    [Crossref] [Google Scholar]
  29. Field JG, Bosco FA, Pierce CA 2013. Variability in effect-size magnitude as a function of sample type. Paper presented at the Annual Meeting of the Academy of Management Lake Buena Vista, FL: Aug. 9–13
  30. Field JG, Mihm DC, O'Boyle EH, Bosco FA, Uggerslev KL, Steel P. 2015. An examination of the funding-finding relation in the field of management Paper presented at the 17th Annual Meeting of the Academy of Management: Opening Governance Vancouver, BC: Aug. 7–11
  31. Furnas GW, Landauer TK, Gomez LM, Dumais ST. 1987. The vocabulary problem in human-system communication. Commun. ACM 30:11964–71
    [Google Scholar]
  32. Gene Ontology Consortium 2015. Gene ontology consortium: going forward. Nucleic Acids Res. 43:D1049–1056
    [Google Scholar]
  33. Gordon ME, Slade LA, Schmitt N. 1986. The “science of the sophomore” revisited: from conjecture to empiricism. Acad. Manag. Rev. 11:191–207
    [Google Scholar]
  34. Götz M, O'Boyle EH, Gonzalez-Mulé E, Banks GC, Bollmann SS. 2021. The “Goldilocks Zone”: (Too) many confidence intervals in tests of mediation just exclude zero. Psychol. Bull. 147:195–114
    [Google Scholar]
  35. Greco LM, O'Boyle EH, Cockburn BS, Yuan Z. 2018. Meta-analysis of coefficient alpha: a reliability generalization study. J. Manag. Stud. 55:583–618
    [Google Scholar]
  36. Hambrick DC. 2007. The field of management's devotion to theory: Too much of a good thing?. Acad. Manag. J. 50:61346–52
    [Google Scholar]
  37. Hardwicke TE, Serghiou S, Janiaud P, Danchev V, Crüwell S, Goodman SN, Ioannidis JP. 2020. Calibrating the scientific ecosystem through meta-research. Annu. Rev. Stat. Appl. 7:11–37
    [Google Scholar]
  38. Hastings J, Frishkoff GA, Smith B, Jensen M, Poldrack RA et al. 2014. Interdisciplinary perspectives on the development, integration, and application of cognitive ontologies. Front. Neuroinformatics 8:62
    [Google Scholar]
  39. He Y, Wang Y, Payne SC 2019. How is safety climate formed? A meta-analysis of the antecedents of safety climate. Organ. Psychol. Rev. 9:2–3124–56
    [Google Scholar]
  40. Head ML, Holman L, Lanfear R, Kahn AT, Jennions MD. 2015. The extent and consequences of p-hacking in science. PLOS Biol 13:3e1002106
    [Google Scholar]
  41. Hofstede G. 1980. Culture's Consequences: International Differences in Work-Related Values Beverly Hills, CA: Sage
  42. House RJ, Hanges PJ, Javidan M, Dorfman PW, Gupta V. 2004. Culture, Leadership, and Organizations Thousand Oaks, CA: Sage
  43. Hutchison KA, Balota DA, Neely JH, Cortese MJ, Cohen-Shikora ER et al. 2013. The semantic priming project. Behav. Res. Methods 45:41099–1114
    [Google Scholar]
  44. Ioannidis JPA. 2014. Discussion: Why “An estimate of the science-wise false discovery rate and application to the top medical literature” is false. Biostatistics 15:128–36
    [Google Scholar]
  45. Ioannidis JPA. 2018. Meta-research: why research on research matters. PLOS Biol 16:3e2005468
    [Google Scholar]
  46. Ioannidis JPA. 2019. What have we (not) learnt from millions of scientific papers with P values?. Am. Stat. 73:20–25
    [Google Scholar]
  47. John LK, Loewenstein G, Prelec D. 2012. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 23:5524–32
    [Google Scholar]
  48. Judge TA, Thoresen CJ, Bono JE, Patton GK. 2001. The job satisfaction–job performance relationship: a qualitative and quantitative review. Psychol. Bull. 127:3376–407
    [Google Scholar]
  49. Kelley TL. 1927. Interpretation of Educational Measurements Chicago: World Book Co.
  50. Kerr NL. 1998. HARKing: hypothesizing after the results are known. Pers. Soc. Psychol. Rev. 2:3196–217
    [Google Scholar]
  51. Kleine AK, Rudolph CW, Zacher H. 2019. Thriving at work: a meta-analysis. J. Organ. Behav. 40:9–10973–99
    [Google Scholar]
  52. Köhler T, Cortina JM, Kurtessis JN, Gölz M. 2015. Are we correcting correctly? Interdependence of reliabilities in meta-analysis. Organ. Res. Methods 18:355–428
    [Google Scholar]
  53. Laird AR, Lancaster JL, Fox PT. 2005. BrainMap: the social evolution of a human brain mapping database. Neuroinformatics 3:65–78
    [Google Scholar]
  54. Larsen KR, Michie S, Hekler EB, Gibson B, Spruijt-Metz D et al. 2017. Behavior change interventions: the potential of ontologies for advancing science and practice. J. Behav. Med. 40:16–22
    [Google Scholar]
  55. Le H, Schmidt FL, Harter JK, Lauver KJ. 2010. The problem of empirical redundancy of constructs in organizational research: an empirical investigation. Organ. Behav. Hum. Decis. Process. 112:2112–25
    [Google Scholar]
  56. Leavitt K, Mitchell T, Peterson J 2010. Theory pruning: strategies for reducing our dense theoretical landscape. Organ. Res. Methods 13:4644–67
    [Google Scholar]
  57. LeBel EP, McCarthy RJ, Earp BD, Elson M, Vanpeamel W. 2018. A unified framework to quantify the credibility of scientific findings. Adv. Methods Pract. Psychol. Sci. 1:3389–402
    [Google Scholar]
  58. Lee CISG, Bosco FA, Steel P, Uggerslev KL. 2017. A metaBUS enabled meta-analysis of career satisfaction. Career Dev. Int. 22:5565–82
    [Google Scholar]
  59. Lee CISG, Felps W, Baruch Y 2014. Toward a taxonomy of career studies through bibliometric visualization. J. Vocat. Behav. 85:3339–51
    [Google Scholar]
  60. Lewis ML, Braginsky M, Tsuji S, Bergmann C, Piccinini PE et al. 2016. A quantitative synthesis of early language acquisition using meta-analysis. PsyArXiv. https://psyarxiv.com/htsjm/
  61. Linnaeus C. 1735. Systema naturae, sive regna tria naturae systematice proposita per classes, ordines, genera, & species Leiden: de Groot
  62. Mackey JD, McAllister CP, Ellen BP III, Carson JE. 2019. A meta-analysis of interpersonal and organizational workplace deviance research. J. Manag. 47:3597–622
    [Google Scholar]
  63. Marshall IJ, Wallace BC. 2019. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Syst. Rev. 8:1163
    [Google Scholar]
  64. Meyer RD. 2015. Taxonomy of Situations and their Measurement Oxford: Oxford Libr. Psychol https://www.oxfordhandbooks.com/view/10.1093/oxfordhb/9780199935291.001.0001/oxfordhb-9780199935291-e-22
  65. Mischel W. 2008. The toothbrush problem. APS Observer Dec. 1. https://www.psychologicalscience.org/observer/the-toothbrush-problem
    [Google Scholar]
  66. Morris SB, DeShon RP. 2002. Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychol. Methods 7:1105–125
    [Google Scholar]
  67. Morse PJ, Neel R, Todd E, Funder D 2015. Renovating situation taxonomies: exploring the construction and content of fundamental motive situation types. J. Pers. 83:4389–403
    [Google Scholar]
  68. Müller-Wille S, Charmantier I 2012. Natural history and information overload: the case of Linnaeus. Stud. Hist. Philos. Sci C. 43:14–15
    [Google Scholar]
  69. Nuijten MB, Hartgerink CHJ, van Assen MALM, Epskamp S, Wicherts JM. 2016. The prevalence of statistical reporting errors in psychology (1985–2013). Behav. Res. Methods 48:1205–26
    [Google Scholar]
  70. Nuijten MB, van Assen MALM, Hartgerink CHJ, Epskamp S, Wicherts JM 2017. The validity of the tool “statcheck” in discovering statistical reporting inconsistencies. PsyArXiv https://psyarxiv.com/tcxaj/
    [Google Scholar]
  71. Parrigon S, Woo SE, Tay L, Wang T. 2017. CAPTION-ing the situation: a lexically-derived taxonomy of psychological situation characteristics. J. Pers. Soc. Psychol. 112:4642–81
    [Google Scholar]
  72. Peterson NG, Mumford MD, Borman WC, Jeanneret PR, Fleishman EA et al. 2001. Understanding work using the Occupational Information Network (O*NET): implications for practice and research. Pers. Psychol. 54:2451–92
    [Google Scholar]
  73. Petrosino A, Boruch RF, Soydan H, Duggan L, Sanchez-Meca J. 2001. Meeting the challenges of evidence-based policy: the Campbell Collaboration. Ann. Am. Acad. Political Soc. Sci. 578:114–34
    [Google Scholar]
  74. Platt JR. 1964. Strong inference. Science 146:3642347–53
    [Google Scholar]
  75. Poldrack RA, Yarkoni T. 2016. From brain maps to cognitive ontologies: informatics and the search for mental structure. Annu. Rev. Psychol. 67:587–612
    [Google Scholar]
  76. Rauthmann JF, Gallardo-Pujol D, Guillaume EM, Todd E, Nave CS et al. 2014. The Situational Eight DIAMONDS: a taxonomy of major dimensions of situation characteristics. J. Pers. Soc. Psychol. 107:4677–718
    [Google Scholar]
  77. Ronen S, Shenkar O 1985. Clustering countries on attitudinal dimensions: a review and synthesis. Acad. Manag. Rev. 10:3435–54
    [Google Scholar]
  78. Rudolph CW, Kooij DT, Rauvola RS, Zacher H. 2018. Occupational future time perspective: a meta-analysis of antecedents and outcomes. J. Organ. Behav. 39:2229–48
    [Google Scholar]
  79. Saran A, White H, Kuper H. 2020. Evidence and gap map of studies assessing the effectiveness of interventions for people with disabilities in low-and middle-income countries. Campbell Syst. Rev. 16:1e1070
    [Google Scholar]
  80. Schmidt T. 2017. Statcheck does not work: all the numbers. Reply to Nuijten et al. 2017. PsyArXiv. https://psyarxiv.com/hr6qy/
  81. Schriger DL, Chehrazi AC, Merchant RM, Altman DG. 2011. Use of the internet by print medical journals in 2003 to 2009: a longitudinal observational study. Ann. Emerg. Med. 57:2153–60
    [Google Scholar]
  82. Shaffer JA, DeGeest D, Li A 2016. Tackling the problem of construct proliferation: a guide to assessing the discriminant validity of conceptually related constructs. Organ. Res. Methods 19:180–110
    [Google Scholar]
  83. Spadaro G, Tiddi I, Columbus S, Jin S, Teije AT, Balliet D 2020. The Cooperation Databank. PsyArXiv. https://doi.org/10.31234/osf.io/rveh3
    [Crossref]
  84. Steel P, Schmidt J, Bosco FA, Uggerslev KL. 2019. The effects of personality on job satisfaction and life satisfaction: a meta-analytic investigation accounting for bandwidth–fidelity and commensurability. Hum. Relat. 72:217–47
    [Google Scholar]
  85. Steel P, Taras V, Uggerslev KL, Bosco FA. 2018. The happy culture: a theoretical, meta-analytic and empirical review of the relationship between culture and wealth and subjective wellbeing. Pers. Soc. Psychol. Rev. 22:128–69
    [Google Scholar]
  86. Tackett JL, Brandes CM, Dworak EM, Shields AN. 2020. Bringing the (pre) registration revolution to graduate training. Can. Psychol./Psychol. Can. 61:299–309
    [Google Scholar]
  87. van Aert RCM, Wicherts JM, van Assen MALM. 2019. Publication bias examined in meta-analyses from psychology and medicine: a meta-meta-analysis. PLOS ONE 14:4e0215052
    [Google Scholar]
  88. Van Iddekinge CH, Arnold JD, Frieder RE, Roth PL 2019. A meta-analysis of the criterion-related validity of prehire work experience. Pers. Psychol. 72:4571–98
    [Google Scholar]
  89. Vanasse TJ, Fox PM, Barron DS, Robertson M, Eickhoff SB, Lancaster JL, Fox PT. 2018. BrainMap VBM: an environment for structural meta-analysis. Hum. Brain Mapp. 39:83308–25
    [Google Scholar]
  90. Veenhoven R. 2009. World Database of Happiness: tool for dealing with the ‘data-deluge. Psychol. Top. 18:221–46
    [Google Scholar]
  91. Venter JC, Remington K, Heidelberg JF, Halpern AL, Rusch D et al. 2004. Environmental genome shotgun sequencing of the Sargasso Sea. Science 304:566766–74
    [Google Scholar]
  92. Wiernik BM, Ones DS. 2018. Ethical employee behaviors in the consensus taxonomy of counterproductive work behaviors. Int. J. Sel. Assess. 26:136–48
    [Google Scholar]
  93. Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M et al. 2016. Comment: The FAIR Guiding Principles for scientific data management and stewardship. Nat. Sci. Data 3:160018
    [Google Scholar]
  94. Woo SE, O'Boyle EH, Spector PE. 2017. Best practices in developing, conducting, and evaluating inductive research. Hum. Resour. Manag. Rev. 27:2255–64
    [Google Scholar]
  95. Woznyj HM, Banks GC, Dunn AM, Berka G, Woehr D. 2020. Re-introducing cognitive complexity: a meta-analysis and agenda for future research. Hum. Perform. 33:11–33
    [Google Scholar]
  96. Yarkoni T, Poldrack RA, Nichols TE, Van Essen DC, Wager TD. 2011. Large-scale automated synthesis of human functional neuroimaging data. Nat. Methods 8:8665–70
    [Google Scholar]
/content/journals/10.1146/annurev-orgpsych-012420-090657
Loading
/content/journals/10.1146/annurev-orgpsych-012420-090657
Loading

Data & Media loading...

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error