Rapid randomized controlled trials have been surprisingly rare in health care quality improvement (QI) and systems interventions. Applying clinical trials methodology QI work brings two distinct fields together, applying the robustness of randomized controlled trials (RCTs) to the practical, operational learnings of the well-established QI field. Rapid trials also add a third element—speed—that enables health care systems to rapidly test multiple variations of an intervention in much the same way that A/B testing is done in the technology sector. When performed well, these rapid trials free researchers and health care systems from the requirement to be correct the first time (because it is low cost and quick to try something else) while offering a standard of evidence often absent in QI. Here we outline the historical underpinnings of this approach, provide guidance about how best to implement it, and describe lessons learned from running more than 20 randomized projects in the NYU Langone Health system.


Article metrics loading...

Loading full text...

Full text loading...


Literature Cited

  1. 1.
    AHRQ (Agency Healthc. Res. Qual.) 2019. Healthcare acquired conditions (HAC) annual report Rep. AHRQ Rockville, MD: https://www.ahrq.gov/hai/pfp/hacreport.html
  2. 2.
    Austrian J, Mendoza F, Szerencsy A, Fenelon L, Horwitz LI et al. 2021. Applying A/B testing to clinical decision support: rapid randomized controlled trials. J. Med. Internet Res. 23:e16651
    [Google Scholar]
  3. 3.
    Baicker K, Taubman SL, Allen HL, Bernstein M, Gruber JH et al. 2013. The Oregon experiment—effects of Medicaid on clinical outcomes. N. Engl. J. Med. 368:1713–22
    [Google Scholar]
  4. 4.
    Banerjee A, Duflo E, Goldberg N, Karlan D, Osei R et al. 2015. A multifaceted program causes lasting progress for the very poor: evidence from six countries. Science 348:1260799
    [Google Scholar]
  5. 5.
    Bulik M. 2016. Which headlines attract most readers?. New York Times June 13. https://www.nytimes.com/2016/06/13/insider/which-headlines-attract-most-readers.html
    [Google Scholar]
  6. 6.
    CDC (Cent. Dis. Control Prev.) 2010. DATA2010 Healthy People 2010 Database, Health Promot. Stat. Branch Hyattsville, MD: reviewed Oct. 31, 2019. http://wonder.cdc.gov/DATA2010/focus.htm
  7. 7.
    CDC (Cent. Dis. Control Prev.) 2019. 2019 national and state healthcare-associated infections progress report Rep. CDC Atlanta: https://www.cdc.gov/hai/data/archive/2019-HAI-progress-report.html
  8. 8.
    Clarke M. 2006. The 1944 patulin trial of the British Medical Research Council. J. R. Soc. Med. 99:478–80
    [Google Scholar]
  9. 9.
    Crofton J. 2006. The MRC randomized trial of streptomycin and its legacy: a view from the clinical front line. J. R. Soc. Med. 99:531–34
    [Google Scholar]
  10. 10.
    Evid.-Based Med. Work. Group 1992. Evidence-based medicine. A new approach to teaching the practice of medicine. JAMA 268:2420–25
    [Google Scholar]
  11. 11.
    Finkelstein A, Zhou A, Taubman S, Doyle J 2020. Health care hotspotting—a randomized, controlled trial. N. Engl. J. Med. 382:152–62
    [Google Scholar]
  12. 12.
    Heck PR, Chabris CF, Watts DJ, Meyer MN. 2020. Objecting to experiments even while approving of the policies or treatments they compare. PNAS 117:18948–50
    [Google Scholar]
  13. 13.
    Horwitz LI, Kuznetsova M, Jones SA. 2019. Creating a learning health system through rapid-cycle, randomized testing. N. Engl. J. Med. 381:1175–79
    [Google Scholar]
  14. 14.
    Inst. Med., Comm. Qualit. Health Care Am 2000. To Err Is Human: Building a Safer Health System LT Kohn, JM Corrigan, MS Donaldson Washington, DC: Natl. Acad. Press
  15. 15.
    Ioannidis JPA. 2005. Contradicted and initially stronger effects in highly cited clinical research. JAMA 294:218–28
    [Google Scholar]
  16. 16.
    Langley GJ, Moen RD, Nolan KM, Nolan TW, Norman CL, Provost LP. 2009. The Improvement Guide: A Practical Approach to Enhancing Organizational Performance San Francisco: Jossey-Bass
  17. 17.
    Lind J. 1753. A Treatise of the Scurvy in Three Parts. Containing an Inquiry into the Nature, Causes, and Cure, of that Disease; Together with A Critical and Chronological View of What Has Been Published on the Subject Edinburgh: A. Kincaid and A. Donaldson
  18. 18.
    Meyer MN, Heck PR, Holtzman GS, Anderson SM, Cai W et al. 2019. Objecting to experiments that compare two unobjectionable policies or treatments. PNAS 116:10723–28
    [Google Scholar]
  19. 19.
    Meyer R. 2014. Everything we know about Facebook's secret mood-manipulation experiment: It was probably legal. But was it ethical?. Atlantic June 28. https://www.theatlantic.com/technology/archive/2014/06/everything-we-know-about-facebooks-secret-mood-manipulation-experiment/373648/
    [Google Scholar]
  20. 20.
    NIH Pragmat. Trials Collab 2022. Demonstration projects. NIH Pragmatic Trials Collaboratory https://rethinkingclinicaltrials.org/demonstration-projects/
    [Google Scholar]
  21. 21.
    NIH Pragmat. Trials Collab 2022. Rethinking clinical trials: a living textbook of pragmatic clinical trials. NIH Pragmatic Trials Collaboratory https://rethinkingclinicaltrials.org/
    [Google Scholar]
  22. 22.
    Off. Natl. Coord. Health Inf. Technol 2017. Non-federal acute care hospital electronic health record adoption. HealthIT.gov. https://dashboard.healthit.gov/quickstats/pages/FIG-Hospital-EHR-Adoption.php
    [Google Scholar]
  23. 23.
    Patulin Clin. Trials Comm. Med. Res. Counc 2004. Clinical trial of patulin in the common cold. 1944. Int. J. Epidemiol. 33:243–46
    [Google Scholar]
  24. 24.
    Pawson R, Greenhalgh T, Harvey G, Walshe K. 2005. Realist review—a new method of systematic review designed for complex policy interventions. J. Health Serv. Res. Policy 10:21–34
    [Google Scholar]
  25. 25.
    Pfadt A, Wheeler DJ. 1995. Using statistical process control to make data-based clinical decisions. J. Appl. Behav. Anal. 28:349–70
    [Google Scholar]
  26. 26.
    Sanbonmatsu L, Ludwig JC, Katz LF, Gennetian LA, Duncan GJ et al. 2011. Moving to Opportunity for fair housing demonstration program—final impacts evaluation Rep. US Dep Hous. Urban Dev. Washington, DC: https://www.huduser.gov/portal/publications/pubasst/MTOFHD.html
  27. 27.
    Sherman LW. 2013. The rise of evidence-based policing: targeting, testing, and tracking. Crime Justice 42:377–451
    [Google Scholar]
  28. 28.
    Shochat SJ, Fremgen AM, Murphy SB, Hutchison C, Donaldson SS et al. 2001. Childhood cancer: patterns of protocol participation in a national survey. CA Cancer J. Clin. 51:119–30
    [Google Scholar]
  29. 29.
    Smith MA, Altekruse SF, Adamson PC, Reaman GH, Seibel NL. 2014. Declining childhood and adolescent cancer mortality. Cancer 120:2497–506
    [Google Scholar]
  30. 30.
    Strauss V. 2018. Pearson conducts experiment on thousands of college students without their knowledge. Washington Post, April 23. https://www.washingtonpost.com/news/answer-sheet/wp/2018/04/23/pearson-conducts-experiment-on-thousands-of-college-students-without-their-knowledge/
    [Google Scholar]
  31. 31.
    Streptomycin Tuberc. Trials Comm. Med. Res. Counc 1948. Streptomycin treatment of tuberculous meningitis. Lancet 251:582–96
    [Google Scholar]
  32. 32.
    Sun Y, Xiong Y, Xu Q, Wei D. 2014. A hadoop-based method to predict potential effective drug combination. BioMed. Res. Int. 2014:196858
    [Google Scholar]
  33. 33.
    Wasserstein RL, Schirm AL, Lazar NA. 2019. Moving to a world beyond “p<0.05.”. Am. Stat. 73:1–19
    [Google Scholar]
  34. 34.
    Wetzel J. 2019. Evidence-Based Practice - Research as responsible practice. Notes from the Field Febr. 15. https://nij.ojp.gov/topics/articles/notes-field-evidence-based-practice-research-responsible-practice
    [Google Scholar]
  35. 35.
    Womack JP, Jones DT, Roos D. 1990. The Machine that Changed the World New York: Free Press

Data & Media loading...

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error