Concerns have been raised in multiple scientific fields in recent years about the reproducibility of published results. Systematic efforts to examine this issue have been undertaken in biomedicine and psychology, but less is known about this important issue in the materials-oriented research that underpins much of modern chemical engineering. Here, we relate a dramatic historical episode from our own institution to illustrate the implications of performing reproducible research and describe two case studies based on literature analysis to provide concrete information on the reproducibility of modern materials-oriented research. The two case studies deal with the properties of metal-organic frameworks (MOFs), a class of materials that have generated tens of thousands of papers. We do not claim that research on MOFs is less (or more) reproducible than other subfields; rather, we argue that the characteristics of this subfield are common to many areas of materials-oriented research. We conclude with specific recommendations for action by individual researchers, journal editors, publishers, and research communities.


Article metrics loading...

Loading full text...

Full text loading...


Literature Cited

  1. 1.
    Harris R. 2017. Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions New York: Basic Books
  2. 2.
    Ioannidis JP. 2005. Why most published research findings are false. PLOS Med 2:e124
    [Google Scholar]
  3. 3.
    Garrison KE, Tang D, Schmeichel BJ 2016. Embodying power: a preregistered replication and extension of the power pose effect. Soc. Psychol. Personal. Sci. 7:623–30
    [Google Scholar]
  4. 4.
    Klein RA, Ratliff KA, Vianello M, Adams R, Bahník Š et al. 2014. Investigating variation in replicability: a “many labs” replication project. Soc. Psychol. 45:142–52
    [Google Scholar]
  5. 5.
    Prinz F, Schlange T, Asadullah K 2011. Believe it or not: How much can we rely on published data on potential drug targets? Nat. Rev. Drug Discov 10:712
    [Google Scholar]
  6. 6.
    Begley CG, Ellis LM. 2012. Drug development: Raise standards for preclinical cancer research. Nature 483:531
    [Google Scholar]
  7. 7.
    Fanelli D. 2018. Opinion: Is science really facing a reproducibility crisis, and do we need it to?. PNAS 115:2628–31
    [Google Scholar]
  8. 8.
    Natl. Renew. Energy Lab 2018. Photovoltaic Research: Device Performance Golden, CO: Natl. Renew. Energy Lab https://www.nrel.gov/pv/device-performance.html
  9. 9.
    Walton KS, Sholl DS. 2017. Research challenges in avoiding “showstoppers” in developing materials for large-scale energy applications. Joule 1:208–11
    [Google Scholar]
  10. 10.
    Mahaffey J. 2017. Atomic Adventures: Secret Islands, Forgotten N-Rays, and Isotopic Murder: A Journey into the Wild World of Nuclear Science New York: Pegasus Books
  11. 11.
    Park J, Howe JD, Sholl DS 2017. How reproducible are isotherm measurements in metal-organic frameworks?. Chem. Mater. 29:10487–95
    [Google Scholar]
  12. 12.
    Eddaoudi M, Kim J, Rosi N, Vodak D, Wachter J et al. 2002. Systematic design of pore size and functionality in isoreticular MOFs and their application in methane storage. Science 295:469–72
    [Google Scholar]
  13. 13.
    Lee J, Farha OK, Roberts J, Scheidt KA, Nguyen ST, Hupp JT 2009. Metal-organic framework materials as catalysts. Chem. Soc. Rev. 38:1450–59
    [Google Scholar]
  14. 14.
    Cohen SM. 2012. Postsynthetic methods for the functionalization of metal-organic frameworks. Chem. Rev. 112:970–1000
    [Google Scholar]
  15. 15.
    Keskin S, van Heest TM, Sholl DS 2010. Can metal-organic framework materials play a useful role in large-scale carbon dioxide separations?. ChemSusChem 3:879–91
    [Google Scholar]
  16. 16.
    Maring BJ, Webley PA. 2013. A new simplified pressure/vacuum swing adsorption model for rapid adsorbent screening for CO2 capture applications. Int. J. Greenh. Gas Control 15:16–31
    [Google Scholar]
  17. 17.
    Nguyen HGT, Espinal L, van Zee RD, Thommes M, Toman B et al. 2018. A reference high-pressure CO2 adsorption isotherm for ammonium ZSM-5 zeolite: results of an interlaboratory study. Adsorption 24:531–39
    [Google Scholar]
  18. 18.
    Siderius DW, Shen VK, Johnson RD III, van Zee RD 2015. NIST/ARPA-E Database of Novel and Emerging Adsorbent Materials Gaithersburg, MD: Natl. Inst. Stand. Technol.
  19. 19.
    Burtch NC, Jasuja H, Walton KS 2014. Water stability and adsorption in metal-organic frameworks. Chem. Rev. 114:10575–612
    [Google Scholar]
  20. 20.
    Groom CR, Allen FH. 2014. The Cambridge Structural Database in retrospect and prospect. Angew. Chem. Int. Ed. 53:662–71
    [Google Scholar]
  21. 21.
    Chung YG, Camp J, Haranczyk M, Sikora BJ, Bury W et al. 2014. Computation-ready, experimental metal-organic frameworks: a tool to enable high-throughput screening of nanoporous crystals. Chem. Mater. 26:6185–92
    [Google Scholar]
  22. 22.
    Volkringer C, Popov D, Loiseau T, Férey G, Burghammer M et al. 2009. Synthesis, single-crystal X-ray microdiffraction, and NMR characterizations of the giant pore metal-organic framework aluminum trimesate MIL-100. Chem. Mater. 21:5695–97
    [Google Scholar]
  23. 23.
    Long P, Wu H, Zhao Q, Wang Y, Dong J, Li J 2011. Solvent effect on the synthesis of MIL-96(Cr) and MIL-100(Cr). Microporous Mesoporous Mater 142:489–93
    [Google Scholar]
  24. 24.
    Zhang B, Wang ZM, Kurmoo M, Gao S, Inoue K, Kobayashi H 2007. Guest-induced chirality in the ferrimagnetic nanoporous diamond framework Mn3(HCOO)6. Adv. Funct. Mater. 17:577–84
    [Google Scholar]
  25. 25.
    Zhang C-Z, Mao H-Y, Wang Y-L, Zhang H-Y, Tao J-C 2007. Syntheses of two new hybrid metal-organic polymers using flexible aliphatic dicarboxylates and pyrazine: crystal structures and magnetic studies. J. Phys. Chem. Solids 68:236–42
    [Google Scholar]
  26. 26.
    Volkringer C, Loiseau T, Férey G, Morais CM, Taulelle F et al. 2007. Synthesis, crystal structure and 71Ga solid state NMR of a MOF-type gallium trimesate (MIL-96) with μ3-oxo bridged trinuclear units and a hexagonal 18-ring network. Microporous Mesoporous Mater 105:111–17
    [Google Scholar]
  27. 27.
    Chang W-M, Cheng M-Y, Liao Y-C, Chang M-C, Wang S-L 2007. Template effect of chain-type polyamines on pore augmentation: five open-framework zinc phosphates with 16-ring channels. Chem. Mater. 19:6114–19
    [Google Scholar]
  28. 28.
    Schull TL, Henley L, Deschamps JR, Butcher RJ, Maher DP et al. 2007. Organometallic supramolecular mixed-valence cobalt(I)/cobalt(II) aquo complexes stabilized with the water-soluble phosphine ligand p-TppTp (p-triphenylphosphine triphosphonic acid). Organometallics 26:2272–76
    [Google Scholar]
  29. 29.
    Kiskin MA, Aleksandrov GG, Bogomyakov AS, Novotortsev VM, Eremenko IL 2008. Coordination polymers of cobalt(II) with pyrimidine and pyrazine: syntheses, structures and magnetic properties. Inorg. Chem. Commun. 11:1015–18
    [Google Scholar]
  30. 30.
    He J, Yang C, Xu Z, Zeller M, Hunter AD, Lin J 2009. Building thiol and metal-thiolate functions into coordination nets: clues from a simple molecule. J. Solid State Chem. 182:1821–26
    [Google Scholar]
  31. 31.
    Konno T, Yoshinari N, Taguchi M, Igashira-Kamiyama A 2009. Drastic change in dimensional structures of D-penicillaminato (AuI2PtI2ZnII)n coordination polymers by moderate change in solution pH. Chem. Lett. 38:526–27
    [Google Scholar]
  32. 32.
    Abrahams BF, Grannas MJ, Hudson TA, Robson R 2010. A simple lithium(I) salt with a microporous structure and its gas sorption properties. Angew. Chem. Int. Ed. 49:1087–89
    [Google Scholar]
  33. 33.
    Hu B-W, Zhao J-P, Tao J, Sun X-J, Yang Q et al. 2010. A new azido-nickel compound with three-dimensional Kagomé topology. Cryst. Growth Design 10:2829–31
    [Google Scholar]
  34. 34.
    Zhang J, Xue Y-S, Liang L-L, Ren S-B, Li Y-Z et al. 2010. Porous coordination polymers of transition metal sulfides with PtS topology built on a semirigid tetrahedral linker. Inorg. Chem. 49:7685–91
    [Google Scholar]
  35. 35.
    Yue Q, Yan L, Zhang J-Y, Gao E-Q 2010. Novel functionalized metal-organic framework based on unique hexagonal prismatic clusters. Inorg. Chem. 49:8647–49
    [Google Scholar]
  36. 36.
    Zou R, Zhong R, Han S, Xu H, Burrell AK et al. 2010. A porous metal-organic replica of α-PbO2 for capture of nerve agent surrogate. J. Am. Chem. Soc. 132:17996–99
    [Google Scholar]
  37. 37.
    Blake AJ, Champness NR, Easun TL, Allan DR, Nowell H et al. 2010. Photoreactivity examined through incorporation in metal-organic frameworks. Nat. Chem. 2:688
    [Google Scholar]
  38. 38.
    An J, Farha OK, Hupp JT, Pohl E, Yeh JI, Rosi NL 2012. Metal-adeninate vertices for the construction of an exceptionally porous metal-organic framework. Nat. Commun. 3:604
    [Google Scholar]
  39. 39.
    Zhang HX, Fu HR, Li HY, Zhang J, Bu X 2013. Porous ctn-type boron imidazolate framework for gas storage and separation. Chemistry 19:11527–30
    [Google Scholar]
  40. 40.
    Natl. Acad. Sci. Eng. Med 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop Washington, DC: Natl. Acad. Press
  41. 41.
    Ioannidis JPA, Greenland S, Hlatky MA, Khoury MJ, Macleod MR et al. 2014. Increasing value and reducing waste in research design, conduct, and analysis. Lancet 383:166–75
    [Google Scholar]
  42. 42.
    Moher D, Glasziou P, Chalmers I, Nasser M, Bossuyt PM et al. 2016. Increasing value and reducing waste in biomedical research: Who's listening?. Lancet 387:1573–86
    [Google Scholar]
  43. 43.
    Bohne C, Liz-Marzán LM, Ganesh KN, Zhang D 2016. Chemistry, from alpha to omega, open to all. ACS Omega 11
  44. 44.
    Torfi A, Iranmanesh SM, Nasrabadi N, Dawson J 2017. 3D convolutional neural networks for cross audio-visual matching recognition. IEEE Access 5:22081–91
    [Google Scholar]
  45. 45.
    Zuilhof H, Yu S-H, Sholl DS 2018. Writing theory and modeling papers for Langmuir: the good, the bad, and the ugly. Langmuir 34:1817–18
    [Google Scholar]

Data & Media loading...

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error