1932
0
  1. Home
  2. A-Z Publications
  3. Annual Review of Public Health
  4. Volume 40, 2019
  5. Article

Annual Review of Public Health

Volume 40, 2019

Making Health Research Matter: A Call to Increase Attention to External Validity

Abstract

Most of the clinical research conducted with the goal of improving health is not generalizable to nonresearch settings. In addition, scientists often fail to replicate each other's findings due, in part, to lack of attention to contextual factors accounting for their relative effectiveness or failure. To address these problems, we review the literature on assessment of external validity and summarize approaches to designing for generalizability. When investigators conduct systematic reviews, a critical need is often unmet: to evaluate the pragmatism and context of interventions, as well as their effectiveness. Researchers, editors, and grant reviewers can implement key changes in how they consider and report on external validity issues. For example, the recently published expanded CONSORT figure may aid scientists and potential program adopters in summarizing participation in and representativeness of a program across different settings, staff, and patients. Greater attention to external validity is needed to increase reporting transparency, improve program dissemination, and reduce failures to replicate research.

Keyword(s): D&I, D4D, designing for dissemination, dissemination and implementation science, external validity, internal validity, replication, translational research
Loading

Article metrics loading...

/content/journals/10.1146/annurev-publhealth-040218-043945
2019-04-01
2025-04-23
Loading full text...

Full text loading...

/deliver/fulltext/publhealth/40/1/annurev-publhealth-040218-043945.html?itemId=/content/journals/10.1146/annurev-publhealth-040218-043945&mimeType=html&fmt=ahah

Literature Cited

  1. 1. 
    Ahmad N, Boutron I, Dechartres A, Durieux P, Ravaud P 2010. Applicability and generalisability of the results of systematic reviews to public health practice and policy: a systematic review. Trials 11:20
     [Google Scholar]
  2. 2. 
    Ammerman A, Smith TW, Calancie L 2014. Practice-based evidence in public health: improving reach, relevance, and results. Annu. Rev. Public Health 35:47–63
     [Google Scholar]
  3. 3. 
    Balas EA, Boren SA 2000. Managing clinical knowledge for health care improvement. Yearb. Med. Inform. 1:65–70
     [Google Scholar]
  4. 4. 
    Boutron I, Altman DG, Moher D, Schulz KF, Ravaud P, CONSORT NPT Group. 2017. CONSORT statement for randomized trials of nonpharmacologic treatments: a 2017 update and a CONSORT extension for nonpharmacologic trial abstracts. Ann. Intern. Med. 167:40–47
     [Google Scholar]
  5. 5. 
    Brown CH, Curran G, Palinkas LA, Aarons GA, Wells KB et al. 2017. An overview of research and evaluation designs for dissemination and implementation. Annu. Rev. Public Health 38:1–22
     [Google Scholar]
  6. 6. 
    Brownson RC, Colditz GA, Proctor EK 2018. Dissemination and Implementation Research in Health: Translating Science to Practice New York: Oxford Univ. Press, 2nd ed..
     [Google Scholar]
  7. 7. 
    Brownson RC, Jacobs JA, Tabak RG, Hoehner CM, Stamatakis KA 2013. Designing for dissemination among public health researchers: findings from a national survey in the United States. Am. J. Public Health 103:1693–99
     [Google Scholar]
  8. 8. 
    Campbell DT, Stanley JC 1963. Experimental and Quasi-Experimental Designs for Research Chicago: R. McNally
     [Google Scholar]
  9. 9. 
    Chambers DA, Glasgow RE, Stange KC 2013. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement. Sci. 8:117
     [Google Scholar]
  10. 10. 
    Chambers DA, Norton WE 2016. The adaptome: advancing the science of intervention adaptation. Am. J. Prev. Med. 51:S124–31
     [Google Scholar]
  11. 11. 
    Chetty R, Stepner M, Abraham S, Lin S, Scuderi B et al. 2016. The association between income and life expectancy in the United States, 2001–2014. JAMA 315:1750–66
     [Google Scholar]
  12. 12. 
    Collins FS 2010. Research agenda. Opportunities for research and NIH. Science 327:36–37
     [Google Scholar]
  13. 13. 
    Community Prev. Serv. Task Force. 2016. Using evidence to improve health outcomes Annu. Rep. Congr., Community Guide Atlanta: https://www.thecommunityguide.org/sites/default/files/assets/2016-congress-report-full.pdf
    [Google Scholar]
  14. 14. 
    Compernolle S, De Cocker K, Lakerveld J, Mackenbach JD, Nijpels G et al. 2014. A RE-AIM evaluation of evidence-based multi-level interventions to improve obesity-related behaviours in adults: a systematic review (the SPOTLIGHT project). Int. J. Behav. Nutr. Phys. Act. 11:147
     [Google Scholar]
  15. 15. 
    Concannon TW, Fuster M, Saunders T, Patel K, Wong JB et al. 2014. A systematic review of stakeholder engagement in comparative effectiveness and patient-centered outcomes research. J. Gen. Intern. Med. 29:1692–701
     [Google Scholar]
  16. 16. 
    Concannon TW, Meissner P, Grunbaum JA, McElwee N, Guise JM et al. 2012. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J. Gen. Intern. Med. 27:985–91
     [Google Scholar]
  17. 17. 
    Cook TD, Campbell DT 1979. Quasi-Experimentation: Design and Analysis Issues for Field Settings Chicago: Rand McNally
     [Google Scholar]
  18. 18. 
    Cronbach LJ, Shapiro K 1982. Designing Evaluations of Educational and Social Programs San Francisco: Jossey-Bass
     [Google Scholar]
  19. 19. 
    Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC 2009. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement. Sci. 4:50
     [Google Scholar]
  20. 20. 
    Daniel H, Bornstein SS, Kane GC, Health Public Policy Comm. Am. Coll. Phys. 2018. Addressing social determinants to improve patient care and promote health equity: an American College of Physicians position paper. Ann. Intern. Med 168:577–78
     [Google Scholar]
  21. 21. 
    De Maeseneer JM, van Driel ML, Green LA, van Weel C 2003. The need for research in primary care. Lancet 362:1314–19
     [Google Scholar]
  22. 22. 
    Eisner EW 2002. The Arts and the Creation of Mind New Haven, CT: Yale Univ. Press
     [Google Scholar]
  23. 23. 
    Feldstein AC, Glasgow RE 2008. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt. Comm. J. Qual. Patient Saf. 34:228–43
     [Google Scholar]
  24. 24. 
    Fisher EB 2008. The importance of context in understanding behavior and promoting health. Ann. Behav. Med. 35:3–18
     [Google Scholar]
  25. 25. 
    Flay BR 1986. Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs. Prev. Med. 15:451–74
     [Google Scholar]
  26. 26. 
    Gaglio B, Phillips SM, Heurtin-Roberts S, Sanchez MA, Glasgow RE 2014. How pragmatic is it? Lessons learned using PRECIS and RE-AIM for determining pragmatic characteristics of research. Implement. Sci. 9:96
     [Google Scholar]
  27. 27. 
    Gaglio B, Shoup JA, Glasgow RE 2013. The RE-AIM framework: a systematic review of use over time. Am. J. Public Health 103:e38–46
     [Google Scholar]
  28. 28. 
    Galea S, Tracy M, Hoggatt KJ, Dimaggio C, Karpati A 2011. Estimated deaths attributable to social factors in the United States. Am. J. Public Health 101:1456–65
     [Google Scholar]
  29. 29. 
    Gartlehner G, Thieda P, Hansen RA, Morgan LC, Shumate JA, Nissman DB 2009. Inadequate reporting of trials compromises the applicability of systematic reviews. Int. J. Technol. Assess. Health Care 25:323–30
     [Google Scholar]
  30. 30. 
    Glasgow RE 2013. What does it mean to be pragmatic? Pragmatic methods, measures, and models to facilitate research translation. Health Educ. Behav. 40:257–65
     [Google Scholar]
  31. 31. 
    Glasgow RE, Estabrooks PE 2018. Pragmatic applications of RE-AIM for health care initiatives in community and clinical settings. Prev. Chronic Dis. 15:170271
     [Google Scholar]
  32. 32. 
    Glasgow RE, Gaglio B, Bennett G, Jerome GJ, Yeh H-C et al. 2012. Applying the PRECIS criteria to describe three effectiveness trials of weight loss in obese patients with comorbid conditions. Health Serv. Res. 47:1051–67
     [Google Scholar]
  33. 33. 
    Glasgow RE, Huebschmann AG, Brownson RC 2018. Expanding the CONSORT figure: increasing transparency in reporting on external validity. Am. J. Prev. Med. 55:422–30
     [Google Scholar]
  34. 34. 
    Glasgow RE, Vogt TM, Boles SM 1999. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am. J. Public Health 89:1322–27
     [Google Scholar]
  35. 35. 
    Glooko. 2018. Glooko API for EHRs and clinical platforms Fact Sheet, Glooko Mountain View, CA: http://get.glooko.com/rs/352-OEL-682/images/PL_API%20for%20Platforms_PRM%200239_16-06_rA.pdf
    [Google Scholar]
  36. 36. 
    Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J et al. 2006. Lost in knowledge translation: time for a map?. J. Contin. Educ. Health Prof. 26:13–24
     [Google Scholar]
  37. 37. 
    Green LW 2006. Public health asks of systems science: to advance our evidence-based practice, can you help us get more practice-based evidence?. Am. J. Public Health 96:406–9
     [Google Scholar]
  38. 38. 
    Green LW, Glasgow RE 2006. Evaluating the relevance, generalization, and applicability of research: issues in external validation and translation methodology. Eval. Health Prof. 29:126–53
     [Google Scholar]
  39. 39. 
    Green LW, Nasser MN 2018. Furthering dissemination and implementation research: the need for more attention to external validity. See Ref. 6301–16Gives an expanded background on the origins of researchers’ bias toward prioritizing internal validity over external validity and a thorough grounding in the domains of external validity that need to be assessed in order to ensure future translation to practice and policy.
  40. 40. 
    Green LW, Ottoson JM, García C, Hiatt RA 2009. Diffusion theory and knowledge dissemination, utilization, and integration in public health. Annu. Rev. Public Health 30:151–74
     [Google Scholar]
  41. 41. 
    Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O 2004. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q 82:581–629
     [Google Scholar]
  42. 42. 
    Harden SM, Gaglio B, Shoup JA, Kinney KA, Johnson SB et al. 2015. Fidelity to and comparative results across behavioral interventions evaluated through the RE-AIM framework: a systematic review. Syst. Rev. 4:155
     [Google Scholar]
  43. 43. 
    Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R et al. 2014. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ 348:g1687
     [Google Scholar]
  44. 44. 
    Holtrop JS, Rabin BA, Glasgow RE 2018. Dissemination and implementation science in primary care research and practice: contributions and opportunities. J. Am. Board Fam. Med. 31:466–78
     [Google Scholar]
  45. 45. 
    Ioannidis JPA 2005. Why most published research findings are false. PLOS Med 2:e124
     [Google Scholar]
  46. 46. 
    Ioannidis JPA 2014. How to make more published research true. PLOS Med 11:e1001747
     [Google Scholar]
  47. 47. 
    Kennedy-Martin T, Curtis S, Faries D, Robinson S, Johnston J 2015. A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results. Trials 16:495
     [Google Scholar]
  48. 48. 
    Kessler RS, Purcell EP, Glasgow RE, Klesges LM, Benkeser RM, Peek CJ 2013. What does it mean to “employ” the RE-AIM model?. Eval. Health Prof. 36:44–66
     [Google Scholar]
  49. 49. 
    Kilbourne AM, Neumann MS, Pincus HA, Bauer MS, Stall R 2007. Implementing evidence-based interventions in health care: application of the replicating effective programs framework. Implement. Sci. 2:42
     [Google Scholar]
  50. 50. 
    Klesges LM, Dzewaltowski DA, Glasgow RE 2008. Review of external validity reporting in childhood obesity prevention research. Am. J. Prev. Med. 34:216–23
     [Google Scholar]
  51. 51. 
    Koppenaal T, Linmans J, Knottnerus JA, Spigt M 2011. Pragmatic vs. explanatory: an adaptation of the PRECIS tool helps to judge the applicability of systematic reviews for daily practice. J. Clin. Epidemiol. 64:1095–101
     [Google Scholar]
  52. 52. 
    Krist AH, Phillips SM, Sabo RT, Balasubramanian BA, Heurtin-Roberts S et al. 2014. Adoption, reach, implementation, and maintenance of a behavioral and mental health assessment in primary care. Ann. Fam. Med. 12:525–33
     [Google Scholar]
  53. 53. 
    Laws RA, St. George AB, Rychetnik L, Bauman AE 2012. Diabetes prevention research: a systematic review of external validity in lifestyle interventions. Am. J. Prev. Med. 43:205–14
     [Google Scholar]
  54. 54. 
    Levine DM, Savarimuthu S, Squires A, Nicholson J, Jay M 2015. Technology-assisted weight loss interventions in primary care: a systematic review. J. Gen. Intern. Med. 30:107–17
     [Google Scholar]
  55. 55. 
    Leviton LC 2017. Generalizing about public health interventions: a mixed-methods approach to external validity. Annu. Rev. Public Health 38:371–91
     [Google Scholar]
  56. 56. 
    Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M 2015. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ 350:h2147
     [Google Scholar]
  57. 57. 
    Luoma KA, Leavitt IM, Marrs JC, Nederveld AL, Regensteiner JG et al. 2017. How can clinical practices pragmatically increase physical activity for patients with type 2 diabetes? A systematic review. Transl. Behav. Med. 7:4751–72
     [Google Scholar]
  58. 58. 
    Manafò E, Petermann L, Vandall-Walker V, Mason-Lai P 2018. Patient and public engagement in priority setting: a systematic rapid review of the literature. PLOS ONE 13:e0193579
     [Google Scholar]
  59. 59. 
    McLeroy KR, Bibeau D, Steckler A, Glanz K 1988. An ecological perspective on health promotion programs. Health Educ. Q. 15:351–77
     [Google Scholar]
  60. 60. 
    Murray CJ, Kulkarni SC, Michaud C, Tomijima N, Bulzacchelli MT et al. 2006. Eight Americas: investigating mortality disparities across races, counties, and race-counties in the United States. PLOS Med 3:e260
     [Google Scholar]
  61. 61. 
    Offringa R, Sheng T, Parks L, Clements M, Kerr D, Greenfield MS 2018. Digital diabetes management application improves glycemic outcomes in people with type 1 and type 2 diabetes. J. Diabetes Sci. Technol. 12:3701–8
     [Google Scholar]
  62. 62. 
    Owen N, Goode A, Sugiyama T, Koohsari MJ, Healy G et al. 2018. Designing for dissemination in chronic disease prevention and management. 6:107–20
     [Google Scholar]
  63. 63. 
    Pedrana L, Pamponet M, Walker R, Costa F, Rasella D 2016. Scoping review: national monitoring frameworks for social determinants of health and health equity. Glob. Health Action 9:28831
     [Google Scholar]
  64. 64. 
    Phimpasone-Brady P, Kwan BM, Holtrop JS, Loskutova N, Rabin BA 2018. Processes, outcomes, and products to make designing for dissemination POP! Presented at the Society of Behavioral Medicine Annual Meeting New Orleans, LA: April 12
    [Google Scholar]
  65. 65. 
    Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G et al. 2017. Standards for Reporting Implementation Studies (StaRI) statement. BMJ 356:i6795
     [Google Scholar]
  66. 66. 
    Rabin BA, Brownson RC, Haire-Joshu D, Kreuter MW, Weaver NL 2008. A glossary for dissemination and implementation research in health. J. Public Health Manag. Pract. 14:117–23
     [Google Scholar]
  67. 67. 
    Rabin BA, Lewis CC, Norton WE, Neta G, Chambers D et al. 2016. Measurement resources for dissemination and implementation research in health. Implement. Sci. 11:42
     [Google Scholar]
  68. 68. 
    Rogers EM 1962. Diffusion of Innovations New York: Free Press of Glencoe
     [Google Scholar]
  69. 69. 
    Rothwell PM 2005. External validity of randomised controlled trials: “To whom do the results of this trial apply?”. Lancet 365:82–93
     [Google Scholar]
  70. 70. 
    Rowling L, Jeffreys V 2006. Capturing complexity: integrating health and education research to inform health-promoting schools policy and practice. Health Educ. Res. 21:705–18
     [Google Scholar]
  71. 71. 
    SAMHSA (Subst. Abuse Ment. Health Serv. Adm.). 2018. Statement of Elinore F. McCance-Katz, MD, PhD, Assistant Secretary for Mental Health and Substance Use regarding the National Registry of Evidence-based Programs and Practices and SAMHSA's new approach to implementation of evidence-based practices (EBPs) Press Release Jan. 11. https://www.samhsa.gov/newsroom/press-announcements/201801110330
    [Google Scholar]
  72. 72. 
    Sanchez MA, Rabin BA, Gaglio B, Henton M, Elzarrad MK et al. 2013. A systematic review of eHealth cancer prevention and control interventions: new technology, same methods and designs?. Transl. Behav. Med. 3:392–401
     [Google Scholar]
  73. 73. 
    Saunders C, Byrne CD, Guthrie B, Lindsay RS, McKnight JA et al. 2013. External validity of randomized controlled trials of glycaemic control and vascular disease: How representative are participants?. Diabet. Med. 30:300–8
     [Google Scholar]
  74. 74. 
    Shippee ND, Allen SV, Leppin AL, May CR, Montori VM 2015. Attaining minimally disruptive medicine: context, challenges and a roadmap for implementation. J. R. Coll. Phys. Edinb. 45:118–22
     [Google Scholar]
  75. 75. 
    Steckler A, McLeroy KR 2008. The importance of external validity. Am. J. Public Health 98:9–10
     [Google Scholar]
  76. 76. 
    Steensma JT, Kreuter MW, Casey CM, Bernhardt JM 2018. Enhancing dissemination through marketing and distribution systems: a vision for public health. 6:191–200
     [Google Scholar]
  77. 77. 
    Stirman SW, Miller CJ, Toder K, Calloway A 2013. Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implement. Sci. 8:65
     [Google Scholar]
  78. 78. 
    Thomson HJ, Thomas S 2012. External validity in healthy public policy: application of the RE-AIM tool to the field of housing improvement. BMC Public Health 12:633
     [Google Scholar]
  79. 79. 
    Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD et al. 2009. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. CMAJ 180:E47–57
     [Google Scholar]
  80. 80. 
    Tomoaia-Cotisel A, Scammon DL, Waitzman NJ, Cronholm PF, Halladay JR et al. 2013. Context matters: the experience of 14 research teams in systematically reporting contextual factors important for practice change. Ann. Fam. Med. 11:Suppl. 1S115–23
     [Google Scholar]
/content/journals/10.1146/annurev-publhealth-040218-043945
Loading
Making Health Research Matter: A Call to Increase Attention to External Validity
Annual Review of Public Health 40, 45 (2019); https://doi.org/10.1146/annurev-publhealth-040218-043945
/content/journals/10.1146/annurev-publhealth-040218-043945
/content/journals/10.1146/annurev-publhealth-040218-043945
Loading

Data & Media loading...

  • Article Type: Review Article

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error