Calibrating the Scientific Ecosystem Through Meta-Research

Literature Cited

1. Alsheikh-Ali AA, Qureshi W, Al-Mallah MH, Ioannidis JPA 2011. Public availability of published research data in high-impact journals. PLOS ONE 6:9e24357
   [Google Scholar]
2. Altman DG. 1994. The scandal of poor medical research. BMJ 308:6924283–84
   [Google Scholar]
3. Altman DG, Simera I. 2016. A history of the evolution of guidelines for reporting medical research: the long road to the EQUATOR Network. J. R. Soc. Med. 109:267–77
   [Google Scholar]
4. Anderson MS, Ronning EA, Devries R, Martinson BC 2010. Extending the Mertonian norms: scientists’ subscription to norms of research. J. High. Educ. 81:3366–93
   [Google Scholar]
5. Armitage P, McPherson CK, Rowe BC 1969. Repeated significance tests on accumulating data. J. R. Stat. Soc. Ser. A. 132:2235–44
   [Google Scholar]
6. Avey MT, Moher D, Sullivan KJ, Fergusson D, Griffin G et al. 2016. The devil is in the details: incomplete reporting in preclinical animal research. PLOS ONE 11:11e0166733
   [Google Scholar]
7. Baker M. 2016. 1,500 scientists lift the lid on reproducibility. Nature 533:7604452–54
   [Google Scholar]
8. Bakker M, van Dijk A, Wicherts JM 2012. The rules of the game called psychological science. Perspect. Psychol. Sci. 7:6543–54
   [Google Scholar]
9. Bakker M, Wicherts JM. 2011. The (mis)reporting of statistical results in psychology journals. Behav. Res. Methods 43:3666–78
   [Google Scholar]
10. Bakker M, Wicherts JM. 2014. Outlier removal, sum scores, and the inflation of the type I error rate in independent samples t tests: the power of alternatives and recommendations. Psychol. Methods 19:3409–27
    [Google Scholar]
11. Barba LA. 2018. Terminologies for reproducible research. arXiv:1802.03311 [cs.DL]
12. Begley CG, Ellis LM. 2012. Raise standards for preclinical cancer research. Nature 483:7391531–33
    [Google Scholar]
13. Benjamin DJ, Berger JO, Johannesson M, Nosek BA, Wagenmakers E-J et al. 2017. Redefine statistical significance. Nat. Hum. Behav. 2:16–10
    [Google Scholar]
14. Bennett CM, Miller MB, Wolford GL 2009. Neural correlates of interspecies perspective taking in the post-mortem Atlantic salmon: an argument for multiple comparisons correction. NeuroImage 47:S1S39–41
    [Google Scholar]
15. Börner K, Klavans R, Patek M, Zoss AM, Biberstine JR et al. 2012. Design and update of a classification system: the UCSD map of science. PLOS ONE 7:7e39464
    [Google Scholar]
16. Boutron I, Dutton S, Ravaud P, Altman DG 2010. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA 303:202058–64
    [Google Scholar]
17. Brown NJL, Heathers JAJ. 2017. The GRIM Test: a simple technique detects numerous anomalies in the reporting of results in psychology. Soc. Psychol. Person. Sci. 8:4363–69
    [Google Scholar]
18. Bruce R, Chauvin A, Trinquart L, Ravuad P, Boutron I 2016. Impact of interventions to improve the quality of peer review of biomedical journals: a systematic review and meta-analysis. BMC Med 14:85
    [Google Scholar]
19. Button KS, Ioannidis JPA, Mokrysz C, Nosek BA, Flint J et al. 2013. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14:5365–76
    [Google Scholar]
20. Camerer CF, Dreber A, Forsell E, Ho T-H, Huber J et al. 2016. Evaluating replicability of laboratory experiments in economics. Science 351:62801433–36
    [Google Scholar]
21. Camerer CF, Dreber A, Holzmeister F, Ho T-H, Huber J et al. 2018. Evaluating the replicability of social science experiments in nature and science between 2010 and 2015. Nat. Hum. Behav. 2:9637–44
    [Google Scholar]
22. Carlisle JB. 2012. The analysis of 168 randomised controlled trials to test data integrity. Anaesthesia 67:5521–37
    [Google Scholar]
23. Carp J. 2012. The secret lives of experiments: methods reporting in the fMRI literature. NeuroImage 63:1289–300
    [Google Scholar]
24. Chalmers I, Glasziou P. 2009. Avoidable waste in the production and reporting of research evidence. Lancet 374:968386–89
    [Google Scholar]
25. Chambers CD. 2013. Registered Reports: a new publishing initiative at Cortex. Cortex 493609–10
    [Google Scholar]
26. Chambers CD, Mellor DT. 2018. Protocol transparency is vital for registered reports. Nat. Hum. Behav. 2:791–92
    [Google Scholar]
27. Chan A-W, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG 2004. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 291:202457–65
    [Google Scholar]
28. Chavalarias D, Ioannidis JPA. 2010. Science mapping analysis characterizes 235 biases in biomedical research. J. Clin. Epidemiol. 63:111205–15
    [Google Scholar]
29. Chavalarias D, Wallach JD, Li AHT, Ioannidis JPA 2016. Evolution of reporting P values in the biomedical literature, 1990–2015. JAMA 315:111141–48
    [Google Scholar]
30. Ciani O, Buyse M, Garside R, Pavey T, Stein K et al. 2013. Comparison of treatment effect sizes associated with surrogate and final patient relevant outcomes in randomised controlled trials: meta-epidemiological study. BMJ 346:f457
    [Google Scholar]
31. Cohen J. 1962. The statistical power of abnormal-social psychological research: a review. J. Abnorm. Soc. Psychol. 65:145–53
    [Google Scholar]
32. Counsell A, Harlow LL. 2017. Reporting practices and use of quantitative methods in Canadian journal articles in psychology. Can. Psychol./Psychol. Can. 58:2140–47
    [Google Scholar]
33. Cramer AOJ, van Ravenzwaaij D, Matzke D, Steingroever H, Wetzels R et al. 2016. Hidden multiplicity in exploratory multiway ANOVA: prevalence and remedies. Psychon. Bull. Rev. 23:2640–47
    [Google Scholar]
34. Cumming G, Fidler F, Leonard M, Kalinowski P, Christiansen A et al. 2007. Statistical reform in psychology: Is anything changing?. Psychol. Sci. 18:3230–32
    [Google Scholar]
35. Dal-Ré R, Ioannidis JPA, Bracken MB, Buffler PA, Chan A-W et al. 2014. Making prospective registration of observational research a reality. Sci. Transl. Med. 6:224224cm1
    [Google Scholar]
36. Dechartres A, Boutron I, Trinquart L, Charles P, Ravaud P 2011. Single-center trials show larger treatment effects than multicenter trials: evidence from a meta-epidemiologic study. Ann. Intern. Med. 155:139–51
    [Google Scholar]
37. Dechartres A, Trinquart L, Atal I, Moher D, Dickersin K et al. 2017. Evolution of poor reporting and inadequate methods over time in 20920 randomised controlled trials included in Cochrane reviews: research on research study. BMJ 357:j2490
    [Google Scholar]
38. Dickersin K, Min YI, Meinert CL 1992. Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. JAMA 267:3374–78
    [Google Scholar]
39. Dickersin K, Rennie D. 2012. The evolution of trial registries and their use to assess the clinical trial enterprise. JAMA 307:171861–64
    [Google Scholar]
40. Dreber A, Pfeiffer T, Almenberg J, Isaksson S, Wilson B et al. 2015. Using prediction markets to estimate the reproducibility of scientific research. PNAS 112:5015343–47
    [Google Scholar]
41. Dwan K, Altman DG, Clarke M, Gamble C, Higgins JPT et al. 2014. Evidence for the selective reporting of analyses and discrepancies in clinical trials: a systematic review of cohort studies of clinical trials. PLOS Med 11:6e1001666
    [Google Scholar]
42. Dwan K, Altman DG, Cresswell L, Blundell M, Gamble CL, Williamson PR 2011. Comparison of protocols and registry entries to published reports for randomised controlled trials. Cochrane Database Syst. Rev. https://doi.org//10.1002/14651858.MR000031.pub2
    [Crossref] [Google Scholar]
43. Dwan K, Gamble C, Williamson PR, Kirkham JJ, Report. Bias Group. 2013. Systematic review of the empirical evidence of study publication bias and outcome reporting bias—an updated review. PLOS ONE 8:7e66844
    [Google Scholar]
44. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR 1991. Publication bias in clinical research. Lancet 337:8746867–72
    [Google Scholar]
45. Elms AC. 1975. The crisis of confidence in social psychology. Am. Psychol. 30:10967–76
    [Google Scholar]
46. Ernst AF, Albers CJ. 2017. Regression assumptions in clinical psychology research practice—a systematic review of common misconceptions. PeerJ 5:e3323
    [Google Scholar]
47. Etz A, Vandekerckhove J. 2016. A Bayesian perspective on the Reproducibility Project: Psychology. PLOS ONE 11:2e0149794
    [Google Scholar]
48. Fanelli D. 2009. How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLOS ONE 4:5e5738
    [Google Scholar]
49. Fanelli D. 2010. Do pressures to publish increase scientists' bias? An empirical support from US states data. PLOS ONE 4:e10271
    [Google Scholar]
50. Fanelli D. 2011. Negative results are disappearing from most disciplines and countries. Scientometrics 90:3891–904
    [Google Scholar]
51. Fanelli D, Costas R, Ioannidis JPA 2017. Meta-assessment of bias in science. PNAS 114:143714–19
    [Google Scholar]
52. Faust D, Meehl PE. 2002. Using meta‐scientific studies to clarify or resolve questions in the philosophy and history of science. Philos. Sci. 69:S3S185–96
    [Google Scholar]
53. Fidler F, Burgman MA, Cumming G, Buttrose R, Thomason N 2006. Impact of criticism of null-hypothesis significance testing on statistical reporting practices in conservation biology. Conserv. Biol. 20:51539–44
    [Google Scholar]
54. Fiedler K. 2011. Voodoo correlations are everywhere—not only in neuroscience. Perspect. Psychol. Sci. 6:2163–71
    [Google Scholar]
55. Forsell E, Viganola D, Pfeiffer T, Almenberg J, Wilson B et al. 2019. Predicting replication outcomes in the Many Labs 2 study. J. Econ. Psych. 75(A):102117
    [Google Scholar]
56. Franco A, Malhotra N, Simonovits G 2014. Publication bias in the social sciences: unlocking the file drawer. Science 345:62031502–5
    [Google Scholar]
57. Franco A, Malhotra N, Simonovits G 2016. Underreporting in psychology experiments. Soc. Psychol. Personal. Sci. 7:18–12
    [Google Scholar]
58. Franzoni C, Scellato G, Stephan P 2011. Changing incentives to publish. Science 333:6043702–3
    [Google Scholar]
59. Fricker RD, Burke K, Han X, Woodall WH 2019. Assessing the statistical analyses used in Basic and Applied Social Psychology after their p-value ban. Am. Stat. 73:1374–84
    [Google Scholar]
60. Gelman A, Loken E. 2014. The statistical crisis in science: data-dependent analysis, a “garden of forking paths,” explains why many statistically significant comparisons don't hold up. Am. Sci. 102:460–65
    [Google Scholar]
61. Gelman A, Stern H. 2006. The difference between “significant” and “not significant” is not itself statistically significant. Am. Stat. 60:328–31
    [Google Scholar]
62. Gigerenzer G. 2004. Mindless statistics. J. Socio-Econ. 33:587–606
    [Google Scholar]
63. Giner-Sorolla R. 2012. Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspect. Psychol. Sci. 7:6562–71
    [Google Scholar]
64. Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M et al. 2014. Reducing waste from incomplete or unusable reports of biomedical research. Lancet 383:9913267–76
    [Google Scholar]
65. Goldacre B, Drysdale H, Dale A, Milosevic I, Slade E et al. 2019. COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time. Trials 20:118
    [Google Scholar]
66. Goodman SN. 1993. p Values, hypothesis tests, and likelihood: implications for epidemiology of a neglected historical debate. Am. J. Epidemiol. 137:5485–96
    [Google Scholar]
67. Goodman SN. 2018. How sure are you of your result? Put a number on it. Nature 564:7
    [Google Scholar]
68. Goodman SN. 2019. Why is getting rid of p-values so hard? Musings on science and statistics. Am. Stat. 73:S126–30
    [Google Scholar]
69. Goodman SN, Fanelli D, Ioannidis JPA 2016. What does research reproducibility mean?. Sci. Transl. Med. 8:3411–6
    [Google Scholar]
70. Gopal AD, Wallach JD, Aminawung JA, Gonsalves G, Dal-Ré R et al. 2018. Adherence to the International Committee of Medical Journal Editors’ (ICMJE) prospective registration policy and implications for outcome integrity: a cross-sectional analysis of trials published in high-impact specialty society journals. Trials 19:1448
    [Google Scholar]
71. Grimes DR, Bauch CT, Ioannidis JPA 2018. Modelling science trustworthiness under publish or perish pressure. R. Soc. Open Sci. 5:1171511
    [Google Scholar]
72. Hardwicke TE, Frank MC, Vazire S, Goodman SN 2019a. Should psychology journals adopt specialized statistical review?. Adv. Methods Pract. Psychol. Sci. https://doi.org/10.1177/2515245919858428
    [Crossref] [Google Scholar]
73. Hardwicke TE, Ioannidis JPA. 2018a. Mapping the universe of registered reports. Nat. Hum. Behav. 2:793–96
    [Google Scholar]
74. Hardwicke TE, Ioannidis JPA. 2018b. Populating the Data Ark: an attempt to retrieve, preserve, and liberate data from the most highly-cited psychology and psychiatry articles. PLOS ONE 13:8e0201856
    [Google Scholar]
75. Hardwicke TE, Mathur MB, MacDonald K, Nilsonne G, Banks GC et al. 2018. Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition. R. Soc. Open Sci 5:8180448
    [Google Scholar]
76. Hardwicke TE, Wallach JD, Kidwell MC, Ioannidis JPA 2019b. An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017). MetaArXiv, Apr. 28. https://doi.org/10.31222/osf.io/6uhg5
    [Crossref]
77. Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JPA 2016. Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: meta-epidemiological survey. BMJ 352:i493
    [Google Scholar]
78. Higginson AD, Munafò MR. 2016. Current incentives for scientists lead to underpowered studies with erroneous conclusions. PLOS Biol 14:11e2000995
    [Google Scholar]
79. Hoekstra R, Finch S, Kiers HAL, Johnson A 2006. Probability as certainty: dichotomous thinking and the misuse of p values. Psychon. Bull. Rev. 13:61033–37
    [Google Scholar]
80. IntHout J, Ioannidis JPA, Borm GF, Goeman JJ 2015. Small studies are more heterogeneous than large ones: a meta-meta-analysis. J. Clin. Epidemiol. 68:8860–69
    [Google Scholar]
81. Ioannidis JPA. 2005. Why most published research findings are false. PLOS Med 2:8e124
    [Google Scholar]
82. Ioannidis JPA. 2008. Why most discovered true associations are inflated. Epidemiology 19:5640–48
    [Google Scholar]
83. Ioannidis JPA. 2012. Why science is not necessarily self-correcting. Perspect. Psychol. Sci. 7:6645–54
    [Google Scholar]
84. Ioannidis JPA. 2014. How to make more published research true. PLOS Med 11:10e1001747
    [Google Scholar]
85. Ioannidis JPA. 2015. Handling the fragile vase of scientific practices. Addiction 110:19–10
    [Google Scholar]
86. Ioannidis JPA. 2016. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q 94:3485–514
    [Google Scholar]
87. Ioannidis JPA. 2017. The reproducibility wars: successful, unsuccessful, uninterpretable, exact, conceptual, triangulated, contested replication. Clin. Chem. 63:5943–45
    [Google Scholar]
88. Ioannidis JPA. 2018a. Meta-research: why research on research matters. PLOS Biol 16:3e2005468
    [Google Scholar]
89. Ioannidis JPA. 2018b. The proposal to lower P value thresholds to.005. JAMA 319:141429–30
    [Google Scholar]
90. Ioannidis JPA. 2019. Retiring statistical significance would give bias a free pass. Nature 567:461
    [Google Scholar]
91. Ioannidis JPA, Caplan AL, Dal-Ré R 2017a. Outcome reporting bias in clinical trials: why monitoring matters. BMJ 356:j408
    [Google Scholar]
92. Ioannidis JPA, Fanelli D, Dunne DD, Goodman SN 2015. Meta-research: evaluation and improvement of research methods and practices. PLOS Biol 13:10e1002264
    [Google Scholar]
93. Ioannidis JPA, Greenland S, Hlatky MA, Khoury MJ, Macleod MR et al. 2014. Increasing value and reducing waste in research design, conduct, and analysis. Lancet 383:9912P166–75
    [Google Scholar]
94. Ioannidis JPA, Stanley TD, Doucouliagos H 2017b. The power of bias in economics research. Econ. J. 127:605F236–65
    [Google Scholar]
95. Ioannidis JPA, Trikalinos TA. 2007. An exploratory test for an excess of significant findings. Clin. Trials. 4:3245–53
    [Google Scholar]
96. Iqbal SA, Wallach JD, Khoury MJ, Schully SD, Ioannidis JPA 2016. Reproducible research practices and transparency across the biomedical literature. PLOS Biol 14:1e1002333
    [Google Scholar]
97. John LK, Loewenstein G, Prelec D 2012. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 23:5524–32
    [Google Scholar]
98. Jones CW, Keil LG, Holland WC, Caughey MC, Platts-Mills TF 2015. Comparison of registered and published outcomes in randomized controlled trials: a systematic review. BMC Med 13:282
    [Google Scholar]
99. Justice AC, Cho MK, Winker MA, Berlin JA, Rennie D 1998. Does masking author identity improve peer review quality? A randomized controlled trial. JAMA 280:3240–42
    [Google Scholar]
100. Kaplan RM, Irvin VL. 2015. Likelihood of null effects of large NHLBI clinical trials has increased over time. PLOS ONE 10:8e0132382
     [Google Scholar]
101. Kerr NL. 1998. HARKing: hypothesizing after the results are known. Personal. Soc. Psychol. Rev. 2:3196–217
     [Google Scholar]
102. Kimmelman J, Mogil JS, Dirnagl U 2014. Distinguishing between exploratory and confirmatory preclinical research will improve translation. PLOS Biol 12:5e1001863
     [Google Scholar]
103. Klein O, Hardwicke TE, Aust F, Breuer J, Danielsson H et al. 2018. A practical guide for transparency in psychological science. Collab. Psychol. 4:120
     [Google Scholar]
104. Klein RA, Vianello M, Hasselman F, Adams BG, Adams RB et al. 2018. Many Labs 2: investigating variation in replicability across samples and settings. Adv. Methods Pract. Psychol. Sci. 1:4443–90
     [Google Scholar]
105. Lane DM, Sándor A. 2009. Designing better graphs by including distributional information and integrating words, numbers, and images. Psychol. Methods 14:3239–57
     [Google Scholar]
106. Lash TL, Vandenbroucke JP. 2012. Should preregistration of epidemiologic study protocols become compulsory?. Epidemiology 23:2184–88
     [Google Scholar]
107. Lazic SE, Clarke-Williams CJ, Munafò MR 2018. What exactly is ‘N’ in cell culture and animal experiments?. PLOS Biol 16:4e2005282
     [Google Scholar]
108. Leamer EE. 1983. Let's take the con out of econometrics. Am. Econ. Rev. 73:131–43
     [Google Scholar]
109. Liddell T, Kruschke JK. 2018. Analyzing ordinal data with metric models: What could possibly go wrong?. J. Exp. Soc. Psychol. 79:328–48
     [Google Scholar]
110. Loder E, Loder S, Cook S 2018. Characteristics and publication fate of unregistered and retrospectively registered clinical trials submitted to the BMJ over 4 years. BMJ Open 8:2e020037
     [Google Scholar]
111. Madigan D, Stang PE, Berlin JA, Schuemie M, Overhage JM et al. 2014. A systematic statistical approach to evaluating evidence from observational studies. Annu. Rev. Stat. Appl. 1:11–39
     [Google Scholar]
112. Makel MC, Plucker JA, Hegarty B 2012. Replications in psychology research. Perspect. Psychol. Sci. 7:537–42
     [Google Scholar]
113. Marwick B, Boettiger C, Mullen L 2017. Packaging data analytical work reproducibly using R (and friends). Am. Stat. 72:180–88
     [Google Scholar]
114. Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P 2009. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA 302:9977–84
     [Google Scholar]
115. Matzke D, Nieuwenhuis S, van Rijn H, Slagter HA, van der Molen MW, Wagenmakers EJ 2015. The effect of horizontal eye movements on free recall: a preregistered adversarial collaboration. J. Exp. Psychol. Gen. 144:1e1–15
     [Google Scholar]
116. Mayo DG. 2018. Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars Cambridge, UK: Cambridge Univ. Press
117. McGillivray B, De Ranieri E 2018. Uptake and outcome of manuscripts in Nature journals by review model and author characteristics. Res. Integr. Peer Rev. 3:5
     [Google Scholar]
118. McShane BB, Gal D, Gelman A, Robert C, Tackett JL 2019. Abandon statistical significance. Am. Stat. 73:1235–45
     [Google Scholar]
119. Merton RK. 1973. The Sociology of Science: Theoretical and Empirical Investigations Chicago: Univ. Chicago Press
120. Meyer MN. 2018. Practical tips for ethical data sharing. Adv. Methods Pract. Psychol. Sci. 1:1131–44
     [Google Scholar]
121. Miguel E, Camerer C, Casey K, Cohen J, Esterling KM et al. 2014. Promoting transparency in social science research. Science 343:616630–31
     [Google Scholar]
122. Moher D, Dulberg CS, Wells GA 1994. Statistical power, sample size, and their reporting in randomized controlled trials. JAMA 272:2122–24
     [Google Scholar]
123. Moher D, Jones A, Lepage L, CONSORT (Consol. Stand. Rep. Trials) Group. 2001. Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. JAMA 285:151992–95
     [Google Scholar]
124. Moher D, Naudet F, Cristea IA, Miedema F, Ioannidis JPA, Goodman SN 2018. Assessing scientists for hiring, promotion, and tenure. PLOS Biol 16:3e2004089
     [Google Scholar]
125. Morey RD, Chambers CD, Etchells PJ, Harris CR, Hoekstra R et al. 2016. The Peer Reviewers’ Openness Initiative: incentivizing open research practices through peer review. R. Soc. Open Sci. 3:1150547
     [Google Scholar]
126. Moshontz H, Campbell L, Ebersole CR, IJzerman H, Urry HL et al. 2018. The Psychological Science Accelerator: advancing psychology through a distributed collaborative network. Adv. Methods Pract. Psychol. Sci. 1:4501–15
     [Google Scholar]
127. Mulkay MJ. 1976. Norms and ideology in science. Soc. Sci. Inf. 15:4–5637–56
     [Google Scholar]
128. Munafò MR, Nosek BA, Bishop BVM, Button KS, Chambers CD et al. 2017. A manifesto for reproducible science. Nat. Hum. Behav. 1:0021
     [Google Scholar]
129. Murad MH, Wang Z. 2017. Guidelines for reporting meta-epidemiological methodology research. Evid. Based Med. 22:4139–42
     [Google Scholar]
130. Naudet F, Sakarovitch C, Janiaud P, Cristea IA, Fanelli D et al. 2018. Data sharing and reanalysis of randomized controlled trials in leading biomedical journals with a full data sharing policy: survey of studies published in the BMJ and PLOS Medicine. . BMJ 360:k400
     [Google Scholar]
131. Nelson LD, Simmons J, Simonsohn U 2018. Psychology's renaissance. Annu. Rev. Psychol. 69:1511–34
     [Google Scholar]
132. Nieuwenhuis S, Forstmann BU, Wagenmakers E-J 2011. Erroneous analyses of interactions in neuroscience: a problem of significance. Nat. Neurosci. 14:91105–7
     [Google Scholar]
133. Nosek BA, Alter G, Banks GC, Borsboom D, Bowman SD et al. 2015. Promoting an open research culture. Science 348:62421422–25
     [Google Scholar]
134. Nosek BA, Ebersole CR, DeHaven AC, Mellor DT 2018. The preregistration revolution. PNAS 115:112600–6
     [Google Scholar]
135. Nosek BA, Errington TM. 2017. Making sense of replications. eLife 6:e23383
     [Google Scholar]
136. Nosek BA, Spies JR, Motyl M 2012. Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspect. Psychol. Sci. 7:6615–31
     [Google Scholar]
137. Nuijten MB, Borghuis J, Veldkamp CLS, Dominguez-Alvarez L, van Assen MALM, Wicherts JM 2017. Journal data sharing policies and statistical reporting inconsistencies in psychology. Collab. Psychol. 3:131
     [Google Scholar]
138. Nuijten MB, Hartgerink CHJ, van Assen MALM, Epskamp S, Wicherts JM 2016. The prevalence of statistical reporting errors in psychology (1985–2013). Behav. Res. Methods 48:41205–26
     [Google Scholar]
139. O'Boyle EH, Banks GC, Gonzalez-Mulé E 2013. The Chrysalis Effect: how ugly data metamorphosize into beautiful articles. Acad. Man. Proc. 43:2376–99
     [Google Scholar]
140. Open Sci. Collab 2015. Estimating the reproducibility of psychological science. Science 349:6251aac4716
     [Google Scholar]
141. Orben A, Przybylski AK. 2019. The association between adolescent well-being and digital technology use. Nat. Hum. Behav. 3:173–82
     [Google Scholar]
142. Page MJ, Higgins JPT, Clayton G, Sterne JAC, Hróbjartsson A, Savović J 2016. Empirical evidence of study design biases in randomized trials: systematic review of meta-epidemiological studies. PLOS ONE 11:7e0159267
     [Google Scholar]
143. Pashler H, Wagenmakers EJ. 2012. Editors’ introduction to the special section on replicability in psychological science: a crisis of confidence?. Perspect. Psychol. Sci. 7:6528–30
     [Google Scholar]
144. Patel CJ, Burford B, Ioannidis JPA 2015. Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations. J. Clin. Epidemiol. 68:91046–58
     [Google Scholar]
145. Pereira TV, Horwitz RI, Ioannidis JPA 2012. Empirical evaluation of very large treatment effects of medical interventions. JAMA 308:161676–84
     [Google Scholar]
146. Phillips CV. 2004. Publication bias in situ. BMC Med. Res. Methods 4:20
     [Google Scholar]
147. Poldrack RA, Baker CI, Durnez J, Gorgolewski KJ, Matthews PM et al. 2017. Scanning the horizon: towards transparent and reproducible neuroimaging research. Nat. Rev. Neurosci. 18:2115–26
     [Google Scholar]
148. Prinz F, Schlange T, Asadullah K 2011. Believe it or not: How much can we rely on published data on potential drug targets?. Nat. Rev. Drug Discov. 10:9712
     [Google Scholar]
149. Robinson KA, Goodman SN. 2011. A systematic examination of the citation of prior research in reports of randomized, controlled trials. Ann. Intern. Med. 154:150–55
     [Google Scholar]
150. Rosenthal R. 1966. Experimenter Effects in Behavioral Research East Norwalk, CT: Appleton-Century-Crofts
151. Rosenthal R. 1979. The file drawer problem and tolerance for null results. Psychol. Bull. 86:3638–41
     [Google Scholar]
152. Ross JS, Tse T, Zarin DA, Xu H, Zhou L, Krumholz HM 2012. Publication of NIH funded trials registered in ClinicalTrials.gov: cross sectional analysis. BMJ 344:d7292
     [Google Scholar]
153. Ross-Hellauer T, Görögh E. 2019. Guidelines for open peer review implementation. Res. Integr. Peer Rev. 4:4
     [Google Scholar]
154. Rowhani-Farid A, Barnett AG. 2016. Has open data arrived at the British Medical Journal (BMJ)? An observational study. BMJ Open 6:10e011784
     [Google Scholar]
155. Sargent RM. 1999. Francis Bacon: Selected Philosophical Works Indianapolis, IN: Hackett
156. Schatz P, Jay KA, McComb J, McLaughlin JR 2005. Misuse of statistical tests in archives of clinical neuropsychology publications. Arch. Clin. Neuropsych. 20:81053–59
     [Google Scholar]
157. Schuemie MJ, Ryan PB, DuMouchel W, Suchard MA, Madigan D 2014. Interpreting observational studies: why empirical calibration is needed to correct p‐values. Stat. Med. 33:2209–18
     [Google Scholar]
158. Scott A, Rucklidge JJ, Mulder RT 2015. Is mandatory prospective trial registration working to prevent publication of unregistered trials and selective outcome reporting? An observational study of five psychiatry journals that mandate prospective clinical trial registration. PLOS ONE 10:8e0133718
     [Google Scholar]
159. Sedlmeier P, Gigerenzer G. 1989. Do studies of statistical power have an effect on the power of studies. Psychol. Bull. 105:2309–16
     [Google Scholar]
160. Seminara D, Khoury MJ, O'Brien TR, Manolio T, Gwinn ML et al. 2007. The emergence of networks in human genome epidemiology: challenges and opportunities. Epidemiology 18:11–8
     [Google Scholar]
161. Shamseer L, Hopewell S, Altman DG, Moher D, Schulz KF 2016. Update on the endorsement of CONSORT by high impact factor journals: a survey of journal “Instructions to Authors” in 2014. Trials 17:1301
     [Google Scholar]
162. Silberzahn R, Uhlmann EL, Martin DP, Anselmi P, Aust F et al. 2018. Many analysts, one data set: making transparent how variations in analytic choices affect results. Adv. Methods Pract. Psychol. Sci. 1:3337–56
     [Google Scholar]
163. Simmons JP, Nelson LD, Simonsohn U 2011. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22:111359–66
     [Google Scholar]
164. Simonsohn U. 2013. Just post it: the lesson from two cases of fabricated data detected by statistics alone. Psychol. Sci. 24:101875–88
     [Google Scholar]
165. Smaldino PE, McElreath R. 2016. The natural selection of bad science. R. Soc. Open Sci. 3:9160384
     [Google Scholar]
166. Spellman BA. 2015. A short (personal) future history of Revolution 2.0. Personal. Psych. Sci. 10:6886–99
     [Google Scholar]
167. Stanley DJ, Spence JR. 2014. Expectations for replications: Are yours realistic. Perspect. Psychol. Sci. 9:3305–18
     [Google Scholar]
168. Steegen S, Tuerlinckx F, Gelman A, Vanpaemel W 2016. Increasing transparency through a multiverse analysis. Perspect. Psychol. Sci. 11:5702–12
     [Google Scholar]
169. Sterling TD. 1959. Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. J. Am. Stat. Assoc. 54:28530–34
     [Google Scholar]
170. Stodden V, Seiler J, Ma Z 2018. An empirical analysis of journal policy effectiveness for computational reproducibility. PNAS 115:112584–89
     [Google Scholar]
171. Strasak AM, Zaman Q, Marinell G, Pfeiffer KP, Ulmer H 2007. The use of statistics in medical research. Am. Stat. 61:147–55
     [Google Scholar]
172. Szucs D, Ioannidis JPA. 2017a. Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLOS Biol 15:3e2000797
     [Google Scholar]
173. Szucs D, Ioannidis JPA. 2017b. When null hypothesis significance testing is unsuitable for research: a reassessment. Front. Hum. Neurosci. 11:390
     [Google Scholar]
174. Tierney JF, Vale C, Riley R, Smith CT, Stewart L et al. 2015. Individual participant data (IPD) meta-analyses of randomised controlled trials: guidance on their use. PLOS Med 12:7e1001855
     [Google Scholar]
175. Trinquart L, Dunn AG, Bourgeois FT 2018. Registration of published randomized trials: a systematic review and meta-analysis. BMC Med 16:1173
     [Google Scholar]
176. Tsakiris M, Martin R, Wagemans J 2018. Re-thinking Cognition’s open data policy: responding to Hardwicke and colleagues' evaluation of its impact. Cognition https://doi.org/10.1016/j.cognition.2018.10.008
     [Crossref] [Google Scholar]
177. Turner L, Shamseer L, Altman DG, Schulz KF, Moher D 2012. Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Syst. Rev. https://doi.org/10.1186/2046-4053-1-60
     [Crossref] [Google Scholar]
178. van Dalen HP, Henkens K 2012. Intended and unintended consequences of a publish‐or‐perish culture: a worldwide survey. J. Am. Soc. Inf. Sci. Tech. 63:71282–93
     [Google Scholar]
179. van Dongen NNN, van Doorn J, Gronau QF, van Ravenzwaaij D, Hoekstra R et al. 2019. Multiple perspectives on inference for two simple statistical scenarios. Am. Stat. 73:S1328–39
     [Google Scholar]
180. Vanpaemel W, Vermorgen M, Deriemaecker L, Storms G 2015. Are we wasting a good crisis? The availability of psychological research data after the storm. Collabra 1:11–5
     [Google Scholar]
181. Vasilevsky NA, Brush MH, Paddock H, Ponting L, Tripathy SJ et al. 2013. On the reproducibility of science: unique identification of research resources in the biomedical literature. PeerJ 1:e148
     [Google Scholar]
182. Vazire S. 2017. Quality uncertainty erodes trust in science. Collab. Psychol. 13:4411–17
     [Google Scholar]
183. Vines TH, Albert AYK, Andrew RL, Débarre F, Bock DG et al. 2014. The availability of research data declines rapidly with article age. Curr. Biol. 24:194–97
     [Google Scholar]
184. Voytek B. 2016. The virtuous cycle of a data ecosystem. PLOS Comput. Biol. 12:8e1005037
     [Google Scholar]
185. Vul E, Harris C, Winkielman P, Pashler H 2009. Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspect. Psychol. Sci. 4:3274–90
     [Google Scholar]
186. Wagenmakers E-J. 2007. A practical solution to the pervasive problems of p values. Psychon. Bull. Rev. 14:5779–804
     [Google Scholar]
187. Wagenmakers E-J, Wetzels R, Borsboom D, van der Maas JLJ, Kievit RA 2012. An agenda for purely confirmatory research. Perspect. Psychol. Sci. 7:6632–38
     [Google Scholar]
188. Walker RL, Sykes L, Hemmelgarn BR, Quan H 2010. Authors’ opinions on publication in relation to annual performance assessment. BMC Med. Educ. 10:21
     [Google Scholar]
189. Wallach JD, Boyack KW, Ioannidis JPA 2018. Reproducible research practices, transparency, and open access data in the biomedical literature, 2015–2017. PLOS Biol 16:11e2006930
     [Google Scholar]
190. Wallach JD, Sullivan PG, Trepanowski JF, Sainani KL, Steyerberg EW, Ioannidis JPA 2017. Evaluation of evidence of statistical support and corroboration of subgroup claims in randomized clinical trials. JAMA Intern. Med. 177:4554–60
     [Google Scholar]
191. Wasserstein RL, Schirm AL, Lazar NA 2019. Moving to a world beyond “p < 0.05. .” Am. Stat 73:S11–19
     [Google Scholar]
192. Weissgerber TL, Garcia-Valencia O, Garovic VD, Milic NM, Winham SJ 2018. Why we need to report more than “data were analyzed by t-tests or ANOVA. .” eLife 7:e36163
     [Google Scholar]
193. Weissgerber TL, Milic NM, Winham SJ, Garovic VD 2015. Beyond bar and line graphs: time for a new data presentation paradigm. PLOS Biol 13:4e1002128
     [Google Scholar]
194. Wicherts JM, Borsboom D, Kats J, Molenaar D 2006. The poor availability of psychological research data for reanalysis. Am. Psychol. 61:7726–28
     [Google Scholar]
195. Wiseman R, Watt C, Kornbrot D 2019. Registered reports: an early example and analysis. PeerJ 7:e6232
     [Google Scholar]
196. World Med. Assoc 2013. World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. JAMA 310:202191–94
     [Google Scholar]
197. Wu L, Wang D, Evans JA 2019. Large teams develop and small teams disrupt science and technology. Nature 566:7744378–82
     [Google Scholar]
198. Yong E. 2012. Replication studies: bad copy. Nature 485:7398298–300
     [Google Scholar]
199. Young NS, Ioannidis JPA, Al-Ubaydli O 2008. Why current publication practices may distort science. PLOS Med 5:10e201
     [Google Scholar]
200. Zarin DA, Tse T, Ide C 2005. Trial registration at ClinicalTrials.gov between May and October 2005. New Engl. J. Med. 353:262779–87
     [Google Scholar]