Replication is a hallmark of science. In recent years, some medical sciences and behavioral sciences struggled with what came to be known as replication crises. As a field, criminology has yet to address formally the threats to our evidence base that might be posed by large-scale and systematic replication attempts, although it is likely we would face challenges similar to those experienced by other disciplines. In this review, we outline the basics of replication, summarize reproducibility problems found in other fields, undertake an original analysis of the amount and nature of replication studies appearing in criminology journals, and consider how criminology can begin to assess more formally the robustness of our knowledge through encouraging a culture of replication.


Article metrics loading...

Loading full text...

Full text loading...


Literature Cited

  1. Am. Soc. Criminol. 2016. Code of Ethics Columbus, OH: Am. Soc. Criminol http://asc41.com/code_of_ethics_copies/ASC_Code_of_Ethics.pdf [Google Scholar]
  2. Anderson CJ, Bahník S, Barnett-Cowan M, Zuni K. 2016. Response to comment on “Estimating the reproducibility of psychological science.”. Science 351:1037 [Google Scholar]
  3. Baker M. 2016. Is there a reproducibility crisis?. Nature 533:453–54 [Google Scholar]
  4. Baumeister RF. 2002. Ego-depletion and self-control failure: an energy model of the self's executive function. Self Identity 1:129–36 [Google Scholar]
  5. Baumeister RF, Bratslavsky E, Muraven M, Tice DM. 1998. Ego depletion: Is the active self a limited resource?. J. Personal. Soc. Psychol. 74:1252–65 [Google Scholar]
  6. Begley CG, Ellis LM. 2012. Raise standards for preclinical cancer research. Nature 483:531–33 [Google Scholar]
  7. Bem D. 2011. Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. J. Personal. Soc. Psychol. 100:407–25 [Google Scholar]
  8. Berk RA, Campbell A, Klap R, Western B. 1992. A Bayesian analysis of the Colorado Springs spouse abuse experiment. J. Crim. Law Criminol. 83:170–200 [Google Scholar]
  9. Brandt MJ, IJzerman H, Dijksterhuis A, Farach F, Geller J. et al. 2014. The replication recipe: What makes for a convincing replication?. J. Exp. Soc. Psychol. 50:217–24 [Google Scholar]
  10. Brodeur A, M, Sangnier M, Zylberberg Y. 2016. Star wars: The empirics strike back. Am. Econ. J. Appl. Econ. 8:1–32 [Google Scholar]
  11. Burger JM. 2009. Replicating Milgram: Would people still obey today?. Am. Psychol. 64:1–11 [Google Scholar]
  12. Camerer CF, Dreber A, Forsell E, Wu H. 2016. Evaluating replicability of laboratory experiments in economics. Science 351:1433–36 [Google Scholar]
  13. Chavalarias D, Wallach JD, Li AH, Ioannidis JP. 2016. Evolution of reporting P values in the biomedical literature. JAMA 315:1141–48 [Google Scholar]
  14. Christakis DA, Zimmerman FJ. 2013. Rethinking reanalysis. JAMA 310:2499–500 [Google Scholar]
  15. Cohen J. 1994. The earth is round (p <.05). Am. Psychol. 49:997–103 [Google Scholar]
  16. Collins HM. 1985. Changing Order: Replication and Induction in Scientific Practice London: Sage [Google Scholar]
  17. Condon DM, Graham E, Mroczek DE. 2017. Replication research. In Wiley-Blackwell Encyclopedia of Personality and Individual Differences BJ Carducci Malden, MA: Wiley-Blackwell. In press [Google Scholar]
  18. de Bono S. 2005. Is the spirit of Piltdown Man alive and well. ? The TelegraphSept. 7 http://www.telegraph.co.uk/technology/3342867/Is-the-spirit-of-Piltdown-man-alive-and-well.html [Google Scholar]
  19. Dolínski D, Grzyb T, Folwarczny M, Trojanowski J. 2017. Would you deliver an electric shock in 2015? Obedience in the experimental paradigm developed by Stanley Milgram in the 50 years following the original studies. Soc. Psychol. Personal. Sci. In press. http://journals.sagepub.com/doi/10.1177/1948550617693060 [Google Scholar]
  20. Donn J. 2016. Medical benefits of dental floss unproven. Associated Press Aug. 2, http://bigstory.ap.org/article/f7e66079d9ba4b4985d7af350619a9e3/medical-benefits-dental-floss-unproven [Google Scholar]
  21. Fanelli D. 2010. “Positive” results increase down the hierarchy of the sciences. PLOS ONE 5:4e10068 [Google Scholar]
  22. Fanelli D. 2012. Negative results are disappearing from most disciplines and countries. Scientometrics 90:891–904 [Google Scholar]
  23. Feynman R, Leighton R. 1985. Surely You're Joking Mr. Feynman! Adventures of a Curious Character. New York: Norton [Google Scholar]
  24. Fleischmann M, Pons S. 1989. Electrochemically induced nuclear fusion of deuterium. J. Electroanal. Chem. 261:301–8 [Google Scholar]
  25. Franco A, Malhotra N, Simonovits G. 2014. Publication bias in the social sciences: unlocking the file drawer. Science 345:1502–5 [Google Scholar]
  26. Freedman LP, Cockburn IM, Simcoe TS. 2015. The economics of reproducibility in preclinical research. PLOS Biol 13:6e1002165 [Google Scholar]
  27. Freese J, Peterson D. 2017. Replication in social science. Annu. Rev. Sociol. 43:147–65 [Google Scholar]
  28. Gilbert DT, King G, Pettigrew S, Wilson TD. 2016. Comment on “Estimating the reproducibility of psychological science.”. Science 351:1037 [Google Scholar]
  29. Grahe JE, Reifman A, Hermann AD, Wiebe RP. 2012. Harnessing the undiscovered resource of student research projects. Perspect. Psychol. Sci. 7:605–7 [Google Scholar]
  30. Guallar E, Stranges S, Mulrow C, Appel LJ, Miller ER III. 2013. Enough is enough: Stop wasting money on vitamin and mineral supplements. Ann. Intern. Med. 159:850–51 [Google Scholar]
  31. Hagger MS, Chatzisarantis NLD, Alberts H, Zwienenberg M. 2016. A multilab preregistered replication of the ego-depletion effect. Perspect. Psychol. Sci. 11:546–73 [Google Scholar]
  32. Hannon L, Defina R. 2016. Can incarceration really strip people of racial privilege. ? Sociol. Sci. 3:190–201 [Google Scholar]
  33. Herndon T, Ash M, Pollin R. 2014. Does high public debt consistently stifle economic growth? A critique of Reinhart & Rogoff. Camb. J. Econ. 38:257–79 [Google Scholar]
  34. Hureau DM. 2016. Essays exploring urban violence PhD Diss., Harvard Univ Cambridge, MA: [Google Scholar]
  35. Hureau DM, Braga AA. 2018. The trade in tools: the market for illicit guns in high-risk networks. Criminology In press [Google Scholar]
  36. Ioannidis JPA. 2005a. Why most published research findings are false. PLOS Med 2:8e124 [Google Scholar]
  37. Ioannidis JPA. 2005b. Contradicted and initially stronger effects in highly cited clinical research. JAMA 294:218–28 [Google Scholar]
  38. Ioannidis JPA. 2008. Why most discovered true associations are inflated. Epidemiology 19:640–48 [Google Scholar]
  39. Iqbal SA, Wallach JD, Khoury MJ, Schully SD, Ioannidis JP. 2016. Reproducible research practices and transparency across the biomedical literature. PLOS Biol 14:1e1002333 [Google Scholar]
  40. John LK, Loewenstein G, Prelec D. 2012. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 23:524–32 [Google Scholar]
  41. Kahneman D. 2014. A new etiquette for replication. Soc. Psychol. 45:310–11 [Google Scholar]
  42. Kidwell MC, Lazaravić LB, Baranski E, Hardwicke TE, Piechowski S. et al. 2016. Badges to acknowledge open practices: a simple, low-cost, effective method for increasing transparency. PLOS Biol 14:5e1002456 [Google Scholar]
  43. Klein RA, Ratliff KA, Vianello M, Adams RB, Bahnik S. et al. 2014. Investigating variation in replicability: a “many labs” replication project. Soc. Psychol. 45:142–52 [Google Scholar]
  44. Kline RB. 2004. Beyond Significance Testing: Reforming Data Analysis Methods in Behavioral Research Washington, DC: Am. Psychol. Assoc. [Google Scholar]
  45. Kunert R. 2016. Internal conceptual replications do not increase independent replication success. Psychon. Bull. Rev. 23:1631–38 [Google Scholar]
  46. Lall R. 2016. How multiple imputation makes a difference. Political Anal 24:414–33 [Google Scholar]
  47. Lauritsen JL. 2005. Explaining patterns of offending across the life course: Comments on interactional theory and recent tests based on the RYDS-RIS data. Ann. Am. Acad. Political Soc. Sci. 602:212–28 [Google Scholar]
  48. LeBel EP. 2015. A new replication norm for psychology. Collabra 1:4 https://www.collabra.org/articles/10.1525/collabra.23/ [Google Scholar]
  49. Leek JT, Peng RD. 2015. Reproducible research can still be wrong: adopting a prevention approach. PNAS 112:1645–46 [Google Scholar]
  50. Lemons CJ, King SA, Davidson KA, Berryessa TL, Gajjar SA, Sacks LH. 2016. An inadvertent concurrent replication. Same roadmap, different journey.. Remedial Spec. Educ. 37:213–22 [Google Scholar]
  51. Lieberson S, Lynn FB. 2002. Barking up the wrong branch: scientific alternatives to the current model of sociological science. Annu. Rev. Sociol. 28:1–19 [Google Scholar]
  52. Long JS. 2009. The Workflow of Data Analysis Using Stata College Station, TX: Stata Press [Google Scholar]
  53. Makel MC, Plucker JA. 2017. Toward a More Perfect Psychology: Improving Trust, Accuracy, and Transparency in Research Washington, DC: Am. Psychol. Assoc [Google Scholar]
  54. Makel MC, Plucker JA. 2014a. Facts are more important than novelty: replication in the education sciences. Educ. Res. 43:304–316 [Google Scholar]
  55. Makel MC, Plucker JA. 2014b. Creativity is more important than novelty: reconsidering replication as a creativity act. Psychol. Aesthet. Creativity Arts 8:27–29 [Google Scholar]
  56. Makel MC, Plucker JA, Freeman J, Lombardi A, Simonsen B, Coyne M. 2016. Replication of special education research: necessary but far too rare. Remedial Spec. Educ. 37:205–12 [Google Scholar]
  57. Makel MC, Plucker JA, Hegarty B. 2012. Replications in psychology research: How often do they really occur. Perspect. Psychol. Sci. 7:537–542 [Google Scholar]
  58. Maniadis Z, Tufano F, List JA. 2014. One swallow doesn't make a summer: new evidence on anchoring effects. Am. Econ. Rev. 104:277–90 [Google Scholar]
  59. McBee M, Makel M, Peters SJ, Matthews MS. 2017. A Manifesto for Open Science in Giftedness Research Charlottesville, VA: Cent. Open Sci https://osf.io/qhwg3/ [Google Scholar]
  60. McNeely S, Warner JJ. 2015. Replication in criminology: necessary practice. Eur. J. Criminol. 12:581–97 [Google Scholar]
  61. Milgram S. 1963. Behavioral study of obedience. J. Abnorm. Soc. Psychol. 67:371–78 [Google Scholar]
  62. Munafo MR, Nosek BA, Bishop DVM, Button KS, Chambers CD. et al. 2017. A manifesto for reproducible science. Nat. Hum. Behav. 1:0021 [Google Scholar]
  63. Neth. Organ. Sci. Res. 2017. Repeating Important Research Thanks to Replication Studies The Hague, Neth.: NWO https://www.nwo.nl/en/news-and-events/news/2017/social-sciences/repeating-important-research-thanks-to-replication-studies.html [Google Scholar]
  64. Nichols T. 2017. The Death of Expertise: The Campaign Against Established Knowledge and Why it Matters New York: Oxford Univ. Press [Google Scholar]
  65. Nosek BA, Alter G, Banks GC, Yarkoni T. 2015. Promoting an open research culture. Science 348:1422–25 [Google Scholar]
  66. Nuijten MB, Hartgerink CH, van Assen MA, Epskamp S, Wicherts JM. 2016. The prevalence of statistical reporting errors in psychology (1985–2013). Behav. Res. Methods 48:1205–26 [Google Scholar]
  67. Nuzzo R. 2015. How scientists fool themselves—and how they can stop. Nature 526:182–85 [Google Scholar]
  68. O'Boyle EH, Banks GC, Gonzalez-Mulé E. 2014. The chrysalis effect: how ugly initial results metamorphosize into beautiful articles. J. Manag. 43:376–99 [Google Scholar]
  69. Open Sci. Collab. 2015. Estimating the reproducibility of psychological science. Science 349:aac4716 [Google Scholar]
  70. Open Sci. Framew. 2017. Course Syllabi for Open and Reproducible Methods Charlottesville, VA: Cent. Open Sci https://osf.io/vkhbt/ [Google Scholar]
  71. Patil PP, Peng RD, Leek JT. 2016. What should researchers expect when they replicate studies? A statistical view of replicability in psychological science. Perspect. Psychol. Sci. 11:539–44 [Google Scholar]
  72. Petrasso RD, Chen X, Wenzel KW, Parker RR, Li CK, Fiore C. 1989. Problems with the γ-ray spectrum in the Fleischmann et al. experiments. Nature 339:183–85 [Google Scholar]
  73. Pickett JT, Roche SP. 2017. Questionable, objectionable or criminal? Public opinion on data fraud and selective reporting in science. Sci. Eng. Ethics. https://doi.org/10.1007/s11948-017-9886-2 [Crossref] [Google Scholar]
  74. Pigott TD, Valentine JC, Polanin JR, Williams RT, Canada DD. 2013. Outcome-reporting bias in education research. Educ. Res. 42:424–32 [Google Scholar]
  75. Pridemore WA. 2011. Poverty matters: a reassessment of the inequality-homicide relationship in cross-national studies. Br. J. Criminol. 51:739–72 [Google Scholar]
  76. Pridemore WA. 2008. A methodological addition to the cross-national empirical literature on social structure and homicide: a first test of the poverty-homicide thesis. Criminology 46:133–54 [Google Scholar]
  77. Prinz F, Schlange T, Asadullah K. 2011. Believe it or not: How much can we rely on published data on potential drug targest?. Nat. Rev. Drug Discov. 10:712–13 [Google Scholar]
  78. Ritchie SJ, Wiseman R, French CC. 2012. Failing the future: Three unsuccessful attempts to replicate Bem's “retroactive facilitation of recall” effect. PLOS ONE 7:3e33423 [Google Scholar]
  79. Sampson RJ, Winship C, Knight C. 2013. Translating causal claims: principles and strategies for policy-relevant criminology. Criminol. Public Policy 12:4587–616 [Google Scholar]
  80. Saperstein A, Penner AM. 2010. The race of a criminal record: how incarceration colors racial perceptions. Soc. Probl. 57:92–113 [Google Scholar]
  81. Schmidt S. 2009. Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Rev. Gen. Psychol. 13:90–100 [Google Scholar]
  82. Schmidt S. 2017. Replication. Toward a More Perfect Psychology: Improving Trust, Accuracy, and Transparency in Research M Makel, JA Plucker 233–54 Washington, DC: Am. Psychol. Assoc. [Google Scholar]
  83. Schwarz N, Clore GL. 2016. Evaluating psychological research requires more than attention to the N: a comment on Simonsohn's (2015) “small telescopes.”. Psychol. Sci. 27:1407–9 [Google Scholar]
  84. Shapin S, Schaffer S. 1985. Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life Princeton, NJ: Princeton Univ. Press [Google Scholar]
  85. Sherman LW, Berk RA. 1984. The specific deterrent effects of arrest for domestic assault. Am. Sociol. Rev. 49:261–72 [Google Scholar]
  86. Sherman LW, Schmidt JD, Rogan DP, Gartin PR, Cohn EG. et al. 1991. From initial deterrence to longterm escalation: short-custody arrest for poverty ghetto domestic violence. Criminology 29:821–50 [Google Scholar]
  87. Simmons JP, Nelson LD, Simonsohn U. 2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22:1359–66 [Google Scholar]
  88. Simmons JP, Nelson LD, Simonsohn U. 2012. A 21 word solution. http://dx.doi.org/10.2139/ssrn.2160588 [Crossref]
  89. Simons DJ, Shoda Y, Lindsay DS. 2017. Constraints on Generality (COG): A Proposed Addition to All Empirical Papers Charlottesville, VA: Cent. Open Sci https://osf.io/phptw/ [Google Scholar]
  90. Simonsohn U. 2015. Small telescopes: detectability and the evaluation of replication results. Psychol. Sci. 26:559–69 [Google Scholar]
  91. Simonsohn U. 2016. Each reader decides if a replication counts: reply to Schwarz and Clore. Psychol. Sci. 27:1410–12 [Google Scholar]
  92. Spellman B, Gilbert EA, Corker KS. 2017. Open Science: What, Why, and How Charlottesville, VA: Cent. Open Sci https://osf.io/8u3qe/
  93. Srivastava S. 2012. A pottery barn rule for scientific journals. The Hardest Science Sept. 27, https://hardsci.wordpress.com/2012/09/27/a-pottery-barn-rule-for-scientific-journals/ [Google Scholar]
  94. Stodden V, McNutt M, Bailey DH, Dellman E, Gil Y. et al. 2016. Enhancing reproducibility for computational methods. Science 354:1240–41 [Google Scholar]
  95. Thornberry TP. 2005. Notes on theory construction and theory testing: a response to Osgood and Lauritsen. Ann. Am. Acad. Political Soc. Sci. 602:229–39 [Google Scholar]
  96. Tittle C. 2004. The arrogance of public sociology. Soc. Forces 82:1639–43 [Google Scholar]
  97. van Elk M, Matzke D, Gronau QF, Guan M, Vanderkerckhove J Wagenmakers E-J. 2015. Meta-analyses are no substitute for registered replications: a skeptical perspective on religious priming. Front. Psychol. 6:1365 [Google Scholar]
  98. Van Noorden R. 2011. The trouble with retractions. Nature 478:26–28 [Google Scholar]
  99. Wellford CF. 2010. Criminologists should stop whining about their impact on policy and practice. Contemporary Issues in Criminal Justice Policy: Policy Proposals from the American Society of Criminology Conference NA Frost, JD Freilich, TR Clear 17–23 Belmont, CA: Wadsworth Cengage Learn. [Google Scholar]
  100. Wicherts JM, Borsboom D, Kats J, Molenaar D. 2006. The poor availability of psychological research data for reanalysis. Am. Psychol. 61:726–28 [Google Scholar]
  101. Wicherts JM, Veldkamp CLS, Augusteijn HEM, Bakker M, van Aert RCM, van Assen MALM. 2016. Degrees of freedom in planning, running, analyzing, and reporting psychological studies: a checklist to avoid p-hacking. Front. Psychol. 7:1832 [Google Scholar]
  102. Young C, Horvath A. 2015. Sociologists need to be better at replication. orgtheory.net Aug. 11, https://orgtheory.wordpress.com/2015/08/11/sociologists-need-to-be-better-at-replication-a-guest-post-by-cristobal-young/ [Google Scholar]

Data & Media loading...

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error