1932
0
  1. Home
  2. A-Z Publications
  3. Annual Review of Statistics and Its Application
  4. Volume 12, 2025
  5. Article

Annual Review of Statistics and Its Application

Volume 12, 2025

Reproducibility in the Classroom

Abstract

Difficulties in reproducing results from scientific studies have lately been referred to as a reproducibility crisis. Scientific practice depends heavily on scientific training. What gets taught in the classroom is often practiced in labs, fieldwork, and data analysis. The importance of reproducibility in the classroom has gained momentum in statistics education in recent years. In this article, we review the existing literature on reproducibility education. We delve into the relationship between computing tools and reproducibility through visiting historical developments in this area. We share examples for teaching reproducibility and reproducible teaching while discussing the pedagogical opportunities created by these examples as well as challenges that the instructors should be aware of. We detail the use of teaching reproducibility and reproducible teaching practices in an introductory data science course. Lastly, we provide recommendations on reproducibility education for instructors, administrators, and other members of the scientific community.

Keyword(s): graduate, literate programming, reproducibility education, reproducible teaching, teaching reproducibility, undergraduate, version control
Loading

Article metrics loading...

/content/journals/10.1146/annurev-statistics-112723-034436
2025-03-07
2025-05-15
Loading full text...

Full text loading...

/deliver/fulltext/statistics/12/1/annurev-statistics-112723-034436.html?itemId=/content/journals/10.1146/annurev-statistics-112723-034436&mimeType=html&fmt=ahah

Literature Cited

  1. Abranovic W, Ageloff R, Frederick D. 1972.. Time-sharing computer systems as a teaching tool. . Am. Stat. 26::34–38
     [Crossref] [Google Scholar]
  2. Adhikari A, DeNero J, Wagner D. 2022.. Computational and Inferential Thinking: The Foundations of Data Science. Online Textbook, Univ. Calif., Berkeley, CA:. https://inferentialthinking.com/chapters/intro.html
    [Google Scholar]
  3. Albert J, Hu J. 2019.. Probability and Bayesian Modeling. Boca Raton, FL:: Chapman and Hall/CRC
     [Google Scholar]
  4. Allaire JJ, Dervieux C. 2024.. quarto: R interface to ‘Quarto’ markdown publishing system. . R Package, version 1.4. https://cran.r-project.org/web/packages/quarto/index.html
    [Google Scholar]
  5. Alperin JP, Schimanski LA, La M, Niles MT, McKiernan EC. 2022.. The value of data and other non-traditional scholarly outputs in academic review, promotion, and tenure in Canada and the United States. . In Open Handbook of Linguistic Data Management, ed. AL Berez-Kroeker, B McDonnell, E Koller, LB Collister , pp. 171–84. Cambridge, MA:: MIT Press
     [Google Scholar]
  6. Amaliah D, Cook D, Tanaka E, Hyde K, Tierney N. 2022.. A journey from wild to textbook data to reproducibly refresh the wages data from the National Longitudinal Survey of Youth database. . J. Stat. Data Sci. Educ. 30::289–303
     [Crossref] [Google Scholar]
  7. Anslow C, Brosz J, Maurer F, Boyes M. 2016.. Datathons: an experience report of data hackathons for data science education. . In Proceedings of the 47th ACM Technical Symposium on Computing Science Education, pp. 615–20. New York:: ACM
     [Google Scholar]
  8. Auer S, Haeltermann NA, Weissgerber TL, Erlich JC, Susilaradeya D, et al. 2021.. A community-led initiative for training in reproducible research. . eLife 10::e64719
     [Crossref] [Google Scholar]
  9. Axfors C, Malički M, Goodman SN. 2024.. Research rigor and reproducibility in research education: a CTSA institutional survey. . J. Clin. Transl. Sci. 8:(1):e45
     [Crossref] [Google Scholar]
  10. Ball R. 2023.. “ Yes we can!”: a practical approach to teaching reproducibility to undergraduates. . Harv. Data Sci. Rev. 5:(3). https://doi.org/10.1162/99608f92.9e002f7b
    [Google Scholar]
  11. Ball R, Medeiros N, Bussberg NW, Piekut A. 2022.. An invitation to teaching reproducible research: lessons from a symposium. . J. Stat. Data Sci. Educ. 30::209–18
     [Crossref] [Google Scholar]
  12. Baumer B, Çetinkaya-Rundel M, Bray A, Loi L, Horton NJ. 2014.. R Markdown: integrating a reproducible analysis tool into introductory statistics. . Technol. Innov. Stat. Educ. 8:(1). http://dx.doi.org/10.5070/T581020118
    [Crossref] [Google Scholar]
  13. Baumer BS, Kaplan DT, Horton NJ. 2021.. Modern Data Science with R. Boca Raton, FL:: Chapman and Hall/CRC
     [Google Scholar]
  14. Bean BL. 2023.. Teaching reproducibility to first year college students: reflections from an introductory data science course. . J. Empower. Teach. Excell. 7:(2):5
     [Google Scholar]
  15. Beckman MD, Çetinkaya-Rundel M, Horton NJ, Rundel CW, Sullivan AJ, Tackett M. 2021.. Implementing version control with Git and GitHub as a learning objective in statistics and data science courses. . J. Stat. Data Sci. Educ. 29::S132–44
     [Crossref] [Google Scholar]
  16. Biehler R. 1997.. Software for learning and for doing statistics. . Int. Stat. Rev. 65::167–69
     [Crossref] [Google Scholar]
  17. Carmer SG, Cady FB. 1969.. Computerized data generation for teaching statistics. . Am. Stat. 23::33–35
     [Crossref] [Google Scholar]
  18. Carver R, Everson M, Gabrosek J, Horton N, Lock R, et al. 2016.. Guidelines for assessment and instruction in statistics education college report 2016. Rep., Am. Stat. Assoc., Alexandria, VA:. https://www.amstat.org/docs/default-source/amstat-documents/gaisecollege_full.pdf
    [Google Scholar]
  19. Chamber JM. 2020.. S, R, and data science. . In Proceedings of the ACM on Programming Languages, Vol. 4, Issue HOPL, art. 84 . New York:: ACM
     [Google Scholar]
  20. Chopik WJ, Bremner RH, Defever AM, Keller VN. 2018.. How (and whether) to teach undergraduates about the replication crisis in psychological science. . Teach. Psychol. 45:(2):158–63
     [Crossref] [Google Scholar]
  21. Chang A, Li P. 2015.. Is economics research replicable? Sixty published papers from thirteen journals say ‘usually not.’ FEDS Work. Pap. 2015-083, Fed. Reserve, Washington, DC:
     [Google Scholar]
  22. Çetinkaya-Rundel M, Diez D, Bray A, Kim AY, Baumer B, et al. 2022.. openintro: Data sets and supplemental functions from ‘OpenIntro’ textbooks and labs. . R Package, version 2.4.0. https://cran.r-project.org/web/packages/openintro/index.html
    [Google Scholar]
  23. Çetinkaya-Rundel M, Dogucu M, Rummerfield W. 2022.. The 5Ws and 1H of term projects in the introductory data science classroom. . Stat. Educ. Res. J. 21:(2):4
     [Crossref] [Google Scholar]
  24. D'Agostino McGowan L. 2023.. Statistical learning. Class Website, Wake Forest Univ., Winston-Salem, NC:. https://sta-363-s23.github.io/website/
    [Google Scholar]
  25. Dogucu M, Çetinkaya-Rundel M. 2022.. Tools and recommendations for reproducible teaching. . J. Stat. Data Sci. Educ. 30::251–60
     [Crossref] [Google Scholar]
  26. Dogucu M, Demirci S, Bendekgey H, Ricci FZ, Medina CM. 2024.. Undergraduate data science education: Who has the microphone and what are they saying?. arXiv:2403.03387 [stat.OT]
  27. Dogucu M, Johnson A, Ott M. 2021.. bayesrules: Datasets and supplemental functions from Bayes Rules! book. . R Package, version 0.0.2. https://cran.r-project.org/web/packages/bayesrules/index.html
    [Google Scholar]
  28. Donoho D. 2017.. 50 years of data science. . J. Comput. Graph. Stat. 26::745–66
     [Crossref] [Google Scholar]
  29. Fanelli D. 2018.. Is science really facing a reproducibility crisis, and do we need it to?. PNAS 115::2628–31
     [Crossref] [Google Scholar]
  30. Fiksel J, Jager LH, Hardin JS, Taub MA. 2019.. Using GitHub classroom to teach statistics. . J. Stat. Data Sci. Educ. 27::110–19
     [Crossref] [Google Scholar]
  31. Fitzgibbon L, Brady D, Haffey A, Kurdi V, Lau J, et al. 2020.. Brewing up a storm: developing open research culture through ReproducibiliTea. Rep., Open Res. Case Stud., Univ. Reading., Reading, UK:. https://doi.org/10.17864/1926.92781
    [Google Scholar]
  32. Frank MC, Saxe R. 2012.. Teaching replication. . Perspect. Psychol. Sci. 7::600–4
     [Crossref] [Google Scholar]
  33. Gould R. 2014.. DataFest: celebrating data in the data deluge. . In Sustainability in Statistics Education: Proceedings of the Ninth International Conference on Teaching Statistics, ed. K Makar, B de Sousa, R Gould . Voorberg, Neth.:: Int. Stat. Inst.
     [Google Scholar]
  34. Halvorsen KT. 2010.. Formulating statistical questions and implementing statistics projects in an introductory applied statistics course. . In Data and Context in Statistics Education: Towards an Evidence-Based Society. Proceedings of the Eighth International Conference on Teaching Statistics, ed. C Reading . Voorberg, Neth.:: Int. Stat. Inst.
     [Google Scholar]
  35. Hettne K, Proppert R, Nab L, Rojas-Saunero LP, Gawehns D. 2020.. ReprohackNL 2019: How libraries can promote research reproducibility through community engagement. . IASSIST Q. 44::1–10
     [Crossref] [Google Scholar]
  36. Hicks S. 2022.. Statistical programming paradigms and workflows. Class Website, Johns Hopkins Univ., Baltimore, MD:. https://www.stephaniehicks.com/jhustatprogramming2022/
    [Google Scholar]
  37. Horton NJ, Alexander R, Parker M, Piekut A, Rundel C. 2022.. Importance of reproducibility and responsible workflow in the data science and statistics curriculum. . J. Stat. Data Sci. Educ. 30::207–8
     [Crossref] [Google Scholar]
  38. Huskey SJ. 2023.. Committing to reproducibility and explainability: using Git as a research journal. . Int. J. Digital Humanit. 6::9–21
     [Crossref] [Google Scholar]
  39. Junk TR, Lyon L. 2021.. Reproducibility and replication of experimental particle physics results. . arXiv:2009.06864 [physics.data-an]
  40. Karathanasis N, Hwang D, Heng V, Abhimannyu R, Slogoff-Sevilla P, et al. 2022.. Reproducibility efforts as a teaching tool: a pilot study. . PLOS Comput. Biol. 18:(11):e1010615
     [Crossref] [Google Scholar]
  41. Kery MB, Radensky M, Arya M, John BE, Myers BA. 2018.. The story in the notebook: exploratory data science using a literate programming tool. . In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pap. 174 . New York:: ACM
     [Google Scholar]
  42. Kluvyer T, Ragan-Kelley B, Pérez F, Granger B, Bussonnier M, et al. 2016.. Jupyter Notebooks – a publishing format for reproducible computational workflows. . In Positioning and Power in Academic Publishing: Players, Agents and Agendas. Proceedings of the 20th International Conference on Electronic Publishing, ed. B Schmidt, F Loizides , pp. 87–90. Amsterdam:: IOS
     [Google Scholar]
  43. Knuth DE. 1984.. Literate programming. . Comput. J. 27::97–111
     [Crossref] [Google Scholar]
  44. Lapane KL, Dube CE. 2021.. Rigor and reproducibility training for first year medical students in research pathways. . Clin. Transl. Sci. 14::1026–36
     [Crossref] [Google Scholar]
  45. LaPolla FWZ, Bakker CJ, Exner N, Montnech T, Surkis A, Ye H. 2022.. Rigor and reproducibility instruction in academic medical libraries. . J. Med. Libr. Assoc. 110::281–93
     [Crossref] [Google Scholar]
  46. Leek JT, Jager LR. 2017.. Is most published research really false?. Annu. Rev. Stat. Appl. 4::109–22
     [Crossref] [Google Scholar]
  47. Legacy C, Zieffler A, Baumer BS, Barr V, Horton NJ. 2023.. Facilitating team-based data science: lessons learned from the DSC-WAV project. . Found. Data Sci. 5::244–65
     [Crossref] [Google Scholar]
  48. Lock R. 2015.. Lock5Data: datasets for “Statistics: Unlocking the Power of Data.”. R Package, version 3.0.0. https://CRAN.R-project.org/package=Lock5Data
    [Google Scholar]
  49. Lucic A, Bleeker M, Jullien S, Bhargav S, De Rijke M. 2022.. Reproducibility as a mechanism for teaching fairness, accountability, confidentiality, and transparency in artificial intelligence. . Proc. AAAI Conf. Artif. Intel. 36:(11):12792–800
     [Google Scholar]
  50. Martonosi SE, Williams TDR. 2016.. A survey of statistical capstone projects. . J. Stat. Educ. 24::127–35
     [Crossref] [Google Scholar]
  51. Marwick B, Wang L, Robinson R, Loiselle H. 2020.. How to use replication assignments for teaching integrity in empirical archaeology. . Adv. Archaeol. Pract. 8::78–86
     [Crossref] [Google Scholar]
  52. McNamara AA. 2015.. Bridging the gap between tools for learning and for doing statistics. PhD Thesis, Dep. Stat., Univ. Calif., Los Angeles:
     [Google Scholar]
  53. Medeiros N, Ball RJ. 2017.. Teaching integrity in empirical economics: the pedagogy of reproducible science in undergraduate education. . In Undergraduate Research and the Academic Librarian: Case Studies and Best Practices, ed. MK Hensley, S Davis-Kahl , pp. 261–68. Chicago:: Am. Libr. Assoc.
     [Google Scholar]
  54. Mehta CC, Moore RH. 2022.. Third time's a charm: a tripartite approach for teaching project organization to students. . J. Stat. Data Sci. Educ. 30::261–65
     [Crossref] [Google Scholar]
  55. Mendez-Carbajo D, Dellachiesa A. 2023.. Data citations and reproducibility in the undergraduate curriculum. . Harv. Data Sci. Rev. 5:(3). https://doi.org/10.1162/99608f92.c2835391
    [Google Scholar]
  56. Milliken G, Nguyen S, Steeves V. 2021.. A behavioral approach to understanding the Git experience. . In Proceedings of the 54th Hawaii International Conference on System Sciences, pp. 7239–48. Honolulu, HI:: Univ. Hawai'i Mānoa
     [Google Scholar]
  57. Millman KJ, Brett M, Barnowski R, Poline JB. 2018.. Teaching computational reproducibility for neuroimaging. . Front. Neurosci. 12:. https://doi.org/10.3389/fnins.2018.00727
    [Crossref] [Google Scholar]
  58. Miyakawa T. 2020.. No raw data, no science: another possible source of the reproducibility crisis. . Mol. Brain 26::745–66
     [Google Scholar]
  59. Natl. Acad. Sci. Eng. Med. 2018.. Data Science for Undergraduates: Opportunities and Options. Washington, DC:: Natl. Acad. Press
     [Google Scholar]
  60. Natl. Acad. Sci. Eng. Med. 2019.. Reproducibility and Replicability in Science. Washington, DC:: Natl. Acad. Press
     [Google Scholar]
  61. Nurse AM, Staiger T. 2019.. Teaching data reproducibility through service learning. . Teach. Sociol. 47::350–57
     [Crossref] [Google Scholar]
  62. Open Sci. Collab. 2015.. Estimating the reproducibility of psychological science. . Science 349::aac4716
     [Crossref] [Google Scholar]
  63. Ostblom J, Timbers T. 2022.. Opinionated practices for teaching reproducibility: motivation, guided instruction and practice. . J. Stat. Data Sci. Educ. 30::241–50
     [Crossref] [Google Scholar]
  64. Peng R. 2015.. The reproducibility crisis in science: a statistical counterattack. . Significance 12::30–32
     [Crossref] [Google Scholar]
  65. Pruim R, Gîrjău MC, Horton NJ. 2023.. Fostering better coding practices for data scientists. . Harv. Data Sci. Rev. 5:(3). https://doi.org/10.1162/99608f92.97c9f60f
    [Google Scholar]
  66. Pruim R, Kaplan DT, Horton NJ. 2024.. mosaic: Project MOSAIC statistics and mathematics teaching utilities. . R Package, version 1.9.1. https://cran.r-project.org/web/packages/mosaic/index.html
    [Google Scholar]
  67. Rethlefsen ML, Lackey MJ, Zhao S. 2018.. Building capacity to encourage research reproducibility and #MakeResearchTrue. . J. Med. Libr. Assoc. 106::113–19
     [Crossref] [Google Scholar]
  68. Ricci FZ, Medina CM, Dogucu M. 2022.. gradetools: Tools to assist with providing grades and personalized feedback to students. . R Package, version 0.2.0. https://github.com/federicazoe/gradetools
    [Google Scholar]
  69. Ricci FZ, Medina CM, Dogucu M. 2024.. Designing and implementing an automated grading workflow for providing personalized feedback to open-ended data science assignments. . Technol. Innov. Stat. Educ. 15:(1). http://dx.doi.org/10.5070/T5.1886
    [Google Scholar]
  70. Roback P, Legler J. 2021.. Beyond Multiple Linear Regression: Applied Generalized Linear Models and Multilevel Models in R. Boca Raton, FL:: Chapman and Hall/CRC
     [Google Scholar]
  71. Rummerfield W, Ricci F, Dogucu M. 2021.. Training graduate students to teach statistics and data science from a distance. . In Statistics Education in the Era of Data Science: Proceedings of the Satellite Conference of the International Association for Statistical Education, ed. R Helenius, E Falck . Voorberg, Neth:.: Int. Stat. Inst.
     [Google Scholar]
  72. Rundel C, Çetinkaya-Rundel M. 2022.. ghclass: Tools for managing classes on GitHub. . R Package, version 0.2.1. https://cran.r-project.org/web/packages/ghclass/index.html
    [Google Scholar]
  73. Sanchez Reyes LL, McTavish EJ. 2022.. Approachable case studies support learning and reproducibility in data science: an example from evolutionary biology. . J. Stat. Data Sci. Educ. 30::304–10
     [Crossref] [Google Scholar]
  74. Seo J, Dogucu M. 2024.. Data science + accessibility. . In Teaching Accessible Computing, ed. A Oleson, AJ Ko, R Ladner . N.p.:: Bookish Press. https://bookish.press/tac/DataScience
    [Google Scholar]
  75. Singer JD, Willett JB. 2003.. Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence. Oxford, UK:: Oxford Univ. Press
     [Google Scholar]
  76. Teal TK, Cranston KA, Lapp H, White E, Wilson G, Ram K, Pawlik A. 2015.. Data carpentry: workshops to increase data literacy for researchers. . Int. J. Digit. Curat. 10::135–43
     [Crossref] [Google Scholar]
  77. Thisted RA. 1979.. Teaching statistical computing using computer packages. . Am. Stat. 33::27–30
     [Crossref] [Google Scholar]
  78. Toelch U, Ostwald D. 2018.. Digital open science—teaching digital tools for reproducible and transparent research. . PLOS Biol. 16::e2006022
     [Crossref] [Google Scholar]
  79. Trisovic A, Lau MK, Pasquier T, Crosas M. 2022.. A large-scale study on research code quality and executions. . Sci. Data 9::60
     [Crossref] [Google Scholar]
  80. Tubb GW, Ringer LJ. 1977.. Current use of computers in the teaching of statistics. Natl. Bur. Stand. Spec. Publ. 503, Gaithersburg, MD:
     [Google Scholar]
  81. Xie Y, Allaire JJ, Grolemund G. 2018.. R Markdown: The Definitive Guide. Boca Raton, FL:: Chapman and Hall/CRC
     [Google Scholar]
  82. Yu B, Hu X. 2019.. Toward training and assessing reproducible data analysis in data science education. . Data Intell. 1::381–92
     [Crossref] [Google Scholar]
/content/journals/10.1146/annurev-statistics-112723-034436
Loading
Reproducibility in the Classroom
Annual Review of Statistics and Its Application 12, 89 (2025); https://doi.org/10.1146/annurev-statistics-112723-034436
/content/journals/10.1146/annurev-statistics-112723-034436
/content/journals/10.1146/annurev-statistics-112723-034436
Loading

Data & Media loading...

  • Article Type: Review Article

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error