1932

Abstract

Across the medical and social sciences, new discussions about replication have led to transformations in research practice. Sociologists, however, have been largely absent from these discussions. The goals of this review are to introduce sociologists to these developments, synthesize insights from science studies about replication in general, and detail the specific issues regarding replication that occur in sociology. The first half of the article argues that a sociologically sophisticated understanding of replication must address both the ways that replication rules and conventions evolved within an epistemic culture and how those cultures are shaped by specific research challenges. The second half outlines the four main dimensions of replicability in quantitative sociology—verifiability, robustness, repeatability, and generalizability—and discusses the specific ambiguities of interpretation that can arise in each. We conclude by advocating some commonsense changes to promote replication while acknowledging the epistemic diversity of our field.

Loading

Article metrics loading...

/content/journals/10.1146/annurev-soc-060116-053450
2017-07-31
2024-03-29
Loading full text...

Full text loading...

/deliver/fulltext/soc/43/1/annurev-soc-060116-053450.html?itemId=/content/journals/10.1146/annurev-soc-060116-053450&mimeType=html&fmt=ahah

Literature Cited

  1. Abascal M, Baldassarri D. 2015. Love thy neighbor? Ethnoracial diversity and trust reexamined. Am. J. Sociol. 121:3722–82 [Google Scholar]
  2. Abbott A. 2007. Notes on replication. Sociol. Methods Res. 36:2210–19 [Google Scholar]
  3. Abramson CM, Dohan D. 2015. Beyond text: using arrays to represent and analyze ethnographic data. Sociol. Methodol. 45:1272–319 [Google Scholar]
  4. Anderson CJ, Bahník S, Barnett-Cowan M, Bosco FA, Chandler J. et al. 2016. Response to comment on “Estimating the reproducibility of psychological science.”. Science 351:62771037 [Google Scholar]
  5. Ashmore M. 1989. The Reflexive Thesis: Wrighting Sociology of Scientific Knowledge Chicago: Univ. Chicago Press
  6. Bahr HM, Caplow T, Chadwick BA. 1983. Middletown III: problems of replication, longitudinal measurement, and triangulation. Annu. Rev. Sociol. 9:243–64 [Google Scholar]
  7. Barro RJ, McCleary RM. 2003. Religion and economic growth across countries. Am. Sociol. Rev. 68:5760–81 [Google Scholar]
  8. Baumeister RF, Bratslavsky E, Muraven M, Tice DM. 1998. Ego depletion: Is the active self a limited resource?. J. Personal. Soc. Psychol. 74:51252–65 [Google Scholar]
  9. Baumeister RF, Vohs KD. 2016. Misguided effort with elusive implications. Perspect. Psychol. Sci. 11:4574–75 [Google Scholar]
  10. Bem D. 2011. Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. J. Personal. Soc. Psychol. 100:407–25 [Google Scholar]
  11. Bernanke BS. 2004. Editorial statement. Am. Econ. Rev. 94:1404 [Google Scholar]
  12. Biernacki R. 2012. Reinventing Evidence in Social Inquiry: Decoding Facts and Variables New York: Palgrave Macmillan
  13. Bosk CL. 2008. What Would You Do? Juggling Bioethics and Ethnography Chicago: Univ. Chicago Press
  14. Bowers J. 2011. Six steps to a better relationship with your future self. Political Methodol 18:22–8 [Google Scholar]
  15. Brandt MJ, IJzerman H, Dijksterhuis A, Farach FJ, Geller J. et al. 2014. The replication recipe: What makes for a convincing replication?. J. Exp. Soc. Psychol. 50:217–24 [Google Scholar]
  16. Breznau N. 2015. The missing main effect of welfare state regimes: a replication of ‘Social policy responsiveness in developed democracies’ by Brooks and Manza. Sociol. Sci. 2:420–41 [Google Scholar]
  17. Brooks C, Manza J. 2006. Social policy responsiveness in developed democracies. Am. Sociol. Rev. 71:3474–94 [Google Scholar]
  18. Brown AN, Cameron DB, Wood BDK. 2014. Quality evidence for policymaking: I'll believe it when I see the replication. J. Dev. Eff. 6:3215–35 [Google Scholar]
  19. Budig MJ, Hodges MJ. 2010. Differences in disadvantage: variation in the motherhood penalty across white women's earnings distributions. Am. Sociol. Rev. 75:5705–28 [Google Scholar]
  20. Budig MJ, Hodges MJ. 2014. Statistical models and empirical evidence for differences in the motherhood penalty across the earnings distribution. Am. Sociol. Rev. 79:2358–64 [Google Scholar]
  21. Cartwright N. 1999. The Dappled World: A Study in the Boundaries of Science Cambridge, UK: Cambridge Univ. Press
  22. Clemens MA. 2015. The meaning of failed replications: a review and proposal. J. Econ. Surv. 31:1326–42 [Google Scholar]
  23. Cole S. 1992. Making Science: Between Nature and Society Cambridge, MA: Harvard Univ. Press
  24. Collins HM. 1974. The TEA set: tacit knowledge and scientific networks. Sci. Stud. 4:2165–86 [Google Scholar]
  25. Collins HM. 1985. Changing Order: Replication and Induction in Scientific Practice Chicago: Univ. Chicago Press
  26. Collins HM. 1991. The meaning of replication and the sciences of economics. Hist. Political Econ. 23:1123–42 [Google Scholar]
  27. Collins HM. 1998. The meaning of data: open and closed evidential cultures in the search for gravitational waves. Am. J. Sociol. 104:2293–338 [Google Scholar]
  28. Collins HM. 2001. Tacit knowledge, trust and the Q of sapphire. Soc. Stud. Sci. 31:171–85 [Google Scholar]
  29. Collins HM. 2016. Reproducibility of experiments: experimenters’ regress, statistical uncertainty principle, and the replication imperative. Reproducibility: Principles, Problems, and Prospects H Atmanspacher, S Maasen 65–82 New York: Wiley [Google Scholar]
  30. Collins HM, Evans R. 2002. The third wave of science studies: studies of expertise and experience. Soc. Stud. Sci. 32:2235–96 [Google Scholar]
  31. Daston L. 1992. Objectivity and the escape from perspective. Soc. Stud. Sci. 22:4597–618 [Google Scholar]
  32. Daston L, Galison P. 2010. Objectivity New York: Zone Books
  33. Desmond M. 2016. Evicted: Poverty and Profit in the American City New York: Crown
  34. Duneier M. 1999. Sidewalk Chicago: Univ. Chicago Press
  35. Edwards P, Mayernik MS, Batcheller AL, Bowker GC, Borgman CL. 2011. Science friction: data, metadata, and collaboration. Soc. Stud. Sci. 41:5667–90 [Google Scholar]
  36. Eich E. 2014. Business not as usual. Psychol. Sci. 25:13–6 [Google Scholar]
  37. Finkel EJ, Eastwick PW, Reis HT. 2015. Best research practices in psychology: illustrating epistemological and pragmatic considerations with the case of relationship science. J. Personal. Soc. Psychol. 108:2275–97 [Google Scholar]
  38. Freeman D. 1983. Margaret Mead and Samoa Cambridge, MA: Harvard Univ. Press
  39. Freeman D. 1999. The Fateful Hoaxing of Margaret Mead: A Historical Analysis of Her Samoan Research Boulder, CO: Westview Press
  40. Freese J, Peterson D. 2016. The emergence of forensic objectivity. SocArXiv: osf.io/preprints/socarxiv/2ft8x
  41. Freese J, Powell B. 2001. Making love out of nothing at all? Null findings and the Trivers-Willard hypothesis. Am. J. Sociol. 106:61776–86 [Google Scholar]
  42. Frow EK. 2012. Drawing a line: setting guidelines for digital image processing in scientific journal articles. Soc. Stud. Sci. 42:3369–92 [Google Scholar]
  43. Galison P, Stump DJ. 1996. The Disunity of Science: Boundaries, Contexts and Power Palo Alto, CA: Stanford Univ. Press
  44. Gilbert DT, King G, Pettigrew S, Wilson TD. 2016a. A response to the reply to our technical comment on ‘Estimating the reproducibility of psychological science.’ Proj. Harvard. http://projects.iq.harvard.edu/files/psychology-replications/files/gkpw_response_to_osc_rebutal.pdf?m=1457031863
  45. Gilbert DT, King G, Pettigrew S, Wilson TD. 2016b. Comment on estimating the reproducibility of psychological science. Science 351:62771037 [Google Scholar]
  46. Goldberg PK. 2015. Report of the Editor: American Economic Review. Am. Econ. Rev. 105:5698–710 [Google Scholar]
  47. Gonzales JE, Cunningham CA. 2015. The promise of pre-registration in psychological research. Psychol. Sci. Agenda. 29:8 http://www.apa.org/science/about/psa/2015/08/pre-registration.aspx [Google Scholar]
  48. Hacking I. 1983. Representing and Intervening: Introductory Topics in the Philosophy of Natural Science New York: Cambridge Univ. Press
  49. Hacking I. 1996. The disunities of the sciences. The Disunity of Science: Boundaries, Contexts and Power P Galison, D Stump 37–74 Palo Alto, CA: Stanford Univ. Press [Google Scholar]
  50. Hagger MS, Chatzisrantis NLD. 2016. A multilab preregistered replication of the ego-depletion effect. Perspect. Psychol. Sci. 11:4546–73 [Google Scholar]
  51. Hammersley M. 1997. Qualitative data archiving: some reflections on its prospects and problems. Sociology 31:1131–42 [Google Scholar]
  52. Herndon T, Ash M, Pollin R. 2014. Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff. Camb. J. Econ. 38:2257–79 [Google Scholar]
  53. Hughes V. 2014. Simple steps aim to solve science's ‘reproducibility problem.’. Spectrum Nov. 17. https://spectrumnews.org/news/simple-steps-aim-to-solve-sciences-reproducibility-problem/
  54. Isaac JC. 2015. For a more public political science. Perspect. Politics 13:2269–83 [Google Scholar]
  55. Janz N. 2015. Leading journal verifies articles before replication–so far, all replications failed. Political Science Replication May 4. https://politicalsciencereplication.wordpress.com/2015/05/04/leading-journal-verifies-articles-before-publication-so-far-all-replications-failed/
  56. Johnson S. 2006. The Ghost Map: The Story of London's Most Terrifying Epidemic London: Penguin
  57. Kahneman D. 2014. A new etiquette for replication. Soc. Psychol. 45:4310–11 [Google Scholar]
  58. Kanazawa S. 2001. Why we love our children. Am. J. Sociol. 106:61761–76 [Google Scholar]
  59. Kennefick D. 2000. Star crushing: theoretical practice and the theoreticians’ regress. Soc. Stud. Sci. 30:15–40 [Google Scholar]
  60. Kerr NL. 1998. HARKing: hypothesizing after the results are known. Personal. Soc. Psychol. B 2:3196–217 [Google Scholar]
  61. Killewald A, Bearak J. 2010. Is the motherhood penalty larger for low-wage women? A comment on quantile regression. Am. Sociol. Rev. 79:2350–57 [Google Scholar]
  62. Kitcher P. 1995. The Advancement of Science: Science Without Legend, Objectivity Without Illusions Oxford, UK: Oxford Univ. Press
  63. Knorr Cetina K. 1999. Epistemic Cultures: How the Sciences Make Knowledge Cambridge, MA: Harvard Univ. Press
  64. LaCour MJ, Green DP. 2014. When contact changes minds: an experiment on transmission of support for gay equality. Science 346:62151366 [Google Scholar]
  65. Lall R. 2016. How multiple imputation makes a difference. Political Anal 24:4414–33 [Google Scholar]
  66. Leahey E. 2008. Overseeing research practice: the case of data editing. Sci. Technol. Hum. Values 33:5605–30 [Google Scholar]
  67. LeBel EP, Peters KR. 2011. Fearing the future of empirical psychology: Bem's 2011 evidence of PSI as a case study of deficiencies in modal research practice. Rev. Gen. Psychol. 15:4371–79 [Google Scholar]
  68. Link BG, Phelan JC. 1995. Social conditions as fundamental causes of disease. J. Health Soc. Behav. 35:80–94 [Google Scholar]
  69. Long JS. 2009. The Workflow of Data Analysis Using Stata College Station, TX: Stata Press
  70. Longo DL, Drazen JM. 2016. Editorial: data sharing. New Engl. J. Med. 374:276–77 [Google Scholar]
  71. Lucas JW, Kevin M, Marek P. 2013. Considerations on the “replication problem” in sociology. Am. Sociol. 44:217–32 [Google Scholar]
  72. Lutfey K, Freese J. 2005. Toward some fundamentals of fundamental causality: socioeconomic status and health in the routine clinic visit for diabetes. Am. J. Sociol. 110:1326–72 [Google Scholar]
  73. Mack RW. 1951. The need for replication research in sociology. Am. Sociol. Rev. 16:193–94 [Google Scholar]
  74. McCullough BD, McGeary KA, Harrison TD. 2006. Lessons from the JMCB archive. J. Money Credit Bank. 38:41093–107 [Google Scholar]
  75. McPherson M, Smith-Lovin L, Brashears ME. 2006a. Social isolation in America: changes in core discussion networks over two decades. Am. Sociol. Rev. 71:3353–75 [Google Scholar]
  76. McPherson M, Smith-Lovin L, Brashears ME. 2006b. Social isolation in America: changes in core discussion networks over two decades. Am. Sociol. Rev 71:3353–75 Erratum 2008. Am. Sociol. Rev 73:61022 [Google Scholar]
  77. Mead M. 1928. Coming of Age in Samoa New York: William Morrow & Company
  78. Meier KJ. 1995. Replication: a view from the streets. PS: Political Sci. Politics 28:3456–59 [Google Scholar]
  79. Merton RK. 1973. The Sociology of Science: Theoretical and Empirical Investigations Chicago: Univ. Chicago Press
  80. Moravcsik A. 2010. Active citation: a precondition for replicable qualitative research. PS: Political Sci. Politics 43:129–35 [Google Scholar]
  81. Mulkay M. 1984. The scientist talks back: a one-act play, with a moral, about replication and reflexivity in sociology. Soc. Stud. Sci. 14:2265–82 [Google Scholar]
  82. Murphy A, Jerolmack C. 2016. Ethnographic masking in an era of data transparency. Contexts Mar. 19. https://contexts.org/blog/ethnographic-masking-in-an-era-of-data-transparency-2/
  83. Nature. 2014. Journals unite for reproducibility. Nature 515:7 [Google Scholar]
  84. Nosek BA, Alter G, Banks GC, Borsboom D, Bowman SD. et al. 2015. Promoting an open research culture. Science 348:62421422–25 [Google Scholar]
  85. Nosek BA, Spies JR, Motyl M. 2012. Scientific utopia: II. Restructuring incentives and practices to promote truth of publishability. Perspect. Psychol. Sci. 7:6615–31 [Google Scholar]
  86. Open Sci. Collab. 2015. Estimating the reproducibility of psychological science. Science 349:943 [Google Scholar]
  87. Panofsky A. 2014. Misbehaving Science: Controversy and the Development of Behavior Genetics Chicago: Univ. Chicago Press
  88. Parish E, Bloom T, Godlee F. 2015. Statins for people at low risk. BMJ 351:h3908 [Google Scholar]
  89. Peterson D. 2015. All that is solid: bench-building at the frontiers of two experimental sciences. Am. Sociol. Rev. 80:61201–25 [Google Scholar]
  90. Peterson RR. 1996a. A re-evaluation of the economic consequences of divorce. Am. Sociol. Rev. 61:6528–36 [Google Scholar]
  91. Peterson RR. 1996b. Statistical errors, faulty conclusions, misguided policy: reply to Weitzman. Am. Sociol. Rev. 61:6539–40 [Google Scholar]
  92. Pickering A. 1995. The Mangle of Practice: Time, Agency, and Science Chicago: Univ. Chicago Press
  93. Pinch TJ. 1979. Normal explanations of the paranormal: the demarcation problem and fraud in parapsychology. Soc. Stud. Sci. 9:3329–48 [Google Scholar]
  94. Popper K. 1959 (1992). The Logic of Scientific Discovery New York: Routledge
  95. Porter TM. 1995. Trust in Numbers: The Pursuit of Objectivity in Science and Public Life Princeton, NJ: Princeton Univ. Press
  96. Putnam RD. 2007. E pluribus unum: diversity and community in the twenty-first century: the 2006 Johan Skytte Prize Lecture. Scan. Political Stud. 30:2137–74 [Google Scholar]
  97. Reinhart CM, Rogoff KS. 2010. Growth in a time of debt. Am. Econ. Rev. 100:5573–78 [Google Scholar]
  98. Ritchie SJ, Wiseman R, French CC. 2012. Failing the future: three unsuccessful attempts to replicate Bem's ‘retroactive facilitation of recall’ effect. PLOS ONE 7:3e33423 [Google Scholar]
  99. Rose AM. 1953. Generalizations in the social sciences. Am. J. Sociol. 59:149–58 [Google Scholar]
  100. Schnall S. 2014. Clean data: statistical artifacts wash out replication efforts. Soc. Psychol. 45:4315–17 [Google Scholar]
  101. Shapin S. 2010. Never Pure: Historical Studies of Science as if It Was Produced by People with Bodies, Situated in Time, Space, Culture, and Society, and Struggling for Credibility and Authority Baltimore, MD: Johns Hopkins Univ. Press
  102. Simons DJ, Holcombe AO, Spellman BA. 2014. An introduction to registered replication reports at Perspectives on Psychological Science. Perspect. Psychol. Sci. 9:5552–55 [Google Scholar]
  103. Simonsohn U, Nelson LD, Simmons JP. 2014. P-curve: a key to the file-drawer problem. J. Exp. Soc. Psychol. 143:2534–47 [Google Scholar]
  104. Small ML. 2009. How many cases do I need? On science and the logic of case selection in field-based research. Ethnography 10:15–38 [Google Scholar]
  105. Smith CM, Papachristos AV. 2016. Trust thy crooked neighbor: multiplexity in Chicago organized crime networks. Am. Sociol. Rev. 81:4644–67 [Google Scholar]
  106. Taylor C. 1985. Philosophical Papers 1 Human Agency and Language Cambridge, UK: Cambridge Univ. Press
  107. Tischler HL. 2007. Introduction to Sociology Belmont, CA: Wadsworth. , 9th ed..
  108. Travis GDL. 1981. Replicating replication? Aspects of the social construction of learning in planarian worms. Soc. Stud. Sci. 11:111–32 [Google Scholar]
  109. Wagenmakers EJ, Wetzels R, Borsboom D, van der Maas HLJ, Kievit RA. 2012. An agenda for purely confirmatory research. Perspect. Psychol. Sci. 7:6632–38 [Google Scholar]
  110. Walker HA, Cohen BP. 1985. Scope statements: imperatives for evaluating theory. Am. Sociol. Rev. 50:3288–301 [Google Scholar]
  111. Weitzman L. 1985. The Divorce Revolution: The Unexpected Social and Economic Consequences for Women and Children in America New York: Free Press
  112. Wicherts JM, Borsboom D, Kats J, Molenaar D. 2006. The poor availability of psychological research data for reanalysis. Am. Psychol. 61:7726–28 [Google Scholar]
  113. Wilson FD, Smoke GL, Martin JD. 1973. The replication problem in sociology: a report and a suggestion. Sociol. Inq. 43:2141–49 [Google Scholar]
  114. Yearley S. 2005. Making Sense of Science: Understanding the Social Study of Science London: SAGE Publ.
  115. Yin R. 2002. Case Study Research Thousand Oaks, CA: SAGE Publ.
  116. Young C. 2009. Model uncertainty in sociological research: an application to religion and economic growth. Am. Sociol. Rev. 74:6380–97 [Google Scholar]
  117. Young C, Horvath A. 2015. Sociologists need to be better at replication. orgtheory.net Aug. 11. https://orgtheory.wordpress.com/2015/08/11/sociologists-need-to-be-better-at-replication-a-guest-post-by-cristobal-young/
  118. Zuckerman H. 1977. Deviant behavior and social control in science. Deviance and Social Change E Sagarin 87–138 Beverly Hills, CA: SAGE Publ. [Google Scholar]
/content/journals/10.1146/annurev-soc-060116-053450
Loading
/content/journals/10.1146/annurev-soc-060116-053450
Loading

Data & Media loading...

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error