The need to provide sound evidence of the costs and benefits of real-world public health interventions has driven advances in the development and analysis of designs other than the controlled trial in which individuals are randomized to an experimental condition. Attention to methodological quality is of critical importance to ensure that any evaluation can accurately answer three fundamental questions: () Has a change occurred, () did the change occur as a result of the intervention, and () is the degree of change significant? A range of alternatives to the individual randomized controlled trial (RCT) can be used for evaluating such interventions, including the cluster RCT, stepped wedge design, interrupted time series, multiple baseline, and controlled prepost designs. The key features and complexities associated with each of these designs are explored.


Article metrics loading...

Loading full text...

Full text loading...


Literature Cited

  1. Adler A. 1.  1992. What Life Could Mean to You Oxford, UK: Oneworld [Google Scholar]
  2. Ajzen I. 2.  1985. From intentions to actions: a theory of planned behaviour. Action Control: From Cognition to Behavior J Kuhl, J Beckmann 11–40 Berlin: Springer-Verlag [Google Scholar]
  3. 3. Aust. Bur. Stat 2006. Australian social trends, 2000, 4102.0. Last updated May 2. http://www.abs.gov.au/ausstats/[email protected]/2f762f95845417aeca25706c00834efa/738d23457b86defbca2570ec000e2f5c!OpenDocument [Google Scholar]
  4. 4. Aust. Inst. Health Welf 2011. 2010 National Drug Strategy Household Survey Report. Drug Stat. Ser. No. 25. Cat. no. PHE 145 Canberra: Aust. Inst. Health Welf http://www.aihw.gov.au/WorkArea/DownloadAsset.aspx?id=10737421314 [Google Scholar]
  5. 5. Aust. Inst. Health Welf 2013. Public Health Expenditure in Australia 2008–2009. Canberra: Aust. Inst. Health Welf http://www.aihw.gov.au/public-health-expenditure/ [Google Scholar]
  6. Bandura A. 6.  1986. Social Foundations of Thought and Action: A Social Cognitive Theory Englewood Cliffs, NJ: Prentice-Hall [Google Scholar]
  7. Baum F. 7.  1998. The New Public Health: An Australian Perspective Melbourne, Aust.: Oxford Univ. Press [Google Scholar]
  8. Biglan A, Ary D, Koehn V, Levings D, Smith S. 8.  et al. 1996. Mobilizing positive reinforcement in communities to reduce youth access to tobacco. Am. J. Community Psychol. 24:625–38 [Google Scholar]
  9. Biglan A, Ary D, Wagenaar AC. 9.  2000. The value of interrupted time-series experiments for community intervention research. Prev. Sci. 1:31–49 [Google Scholar]
  10. Biglan A, Henderson J, Humphrey D, Yasui M, Whisman R. 10.  et al. 1995. Mobilising positive reinforcement to reduce youth access to tobacco. Tob. Control 4:42–48 [Google Scholar]
  11. Brown CA, Lilford RJ. 11.  2006. The stepped wedge trial design: a systematic review. BMC Med. Res. Methodol. 6:54–63 [Google Scholar]
  12. Brown CH, Wyman PA, Guo J, Peña J. 12.  2006. Dynamic wait-listed designs for randomized trials: new designs for prevention of youth suicide. Clin. Trials 3:259–71 [Google Scholar]
  13. Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P. 13.  et al. 2000. Framework for design and evaluation of complex interventions to improve health. BMJ 321:694–96 [Google Scholar]
  14. Capelastegui A, España PP, Quintana JM, Gorordo I, Ortega M. 14.  et al. 2004. Improvement of process-of-care and outcomes after implementing a guideline for the management of community-acquired pneumonia: a controlled before-and-after design study. Clin. Infect. Dis. 39:955–63 [Google Scholar]
  15. 15. CDC (Cent. Dis. Control Prev.) 2011. Vital signs: current cigarette smoking among adults aged ≥18 years—United States, 2005–2010. MMWR 60:1207–12 [Google Scholar]
  16. 16. CDC (Cent. Dis. Control Prev.) 2012. Trends in current cigarette smoking among high school students and adults, United States, 1965–2011. Updated November 8, 2013. http://www.cdc.gov/tobacco/data_statistics/tables/trends/cig_smoking/index.htm [Google Scholar]
  17. 17. Cent. Evidence Based Medicine 2013. Oxford Centre for Evidence-based Medicine—levels of evidence (March 2009). Updated September 16. http://www.cebm.net/?o=1025 [Google Scholar]
  18. Corrigan O. 18.  2003. Empty ethics: the problem with informed consent. Sociol. Health Illn. 25:768–92 [Google Scholar]
  19. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. 19.  2008. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 337:a1655 [Google Scholar]
  20. Crosbie J. 20.  1993. Interrupted time-series analysis with brief single-subject data. J. Consult. Clin. Psychol. 61:966–74 [Google Scholar]
  21. DeVita VT Jr. 21.  2005. The Framework Convention on Tobacco Control. Nat. Clin. Pract. Oncol. 2:171 [Google Scholar]
  22. Dickinson J, Wiggers J, Leeder S, Sanson-Fisher RW. 22.  1989. General practitioners' detection of patients' smoking status. Med. J. Aust. 150:420–22 425–26 [Google Scholar]
  23. Djulbegovic M, Beyth RJ, Neuberger MM, Stoffs TL, Vieweg J. 23.  et al. 2010. Screening for prostate cancer: systematic review and meta-analysis of randomised controlled trials. BMJ 341:c4543 [Google Scholar]
  24. Donabedian A. 24.  1988. The quality of health care: How can it be assessed?. JAMA 260:1743–48 [Google Scholar]
  25. Donner A, Klar N. 25.  2000. Design and Analysis of Cluster Randomization Trials in Health Research London: Arnold Publ. [Google Scholar]
  26. Eldridge S, Kerry S. 26.  2012. A Practical Guide to Cluster Randomised Trials in Health Services Research Chichester, UK: Wiley [Google Scholar]
  27. Erbas B, Bui Q, Huggins R, Harper T, White V. 27.  2006. Investigating the relation between placement of Quit antismoking advertisements and number of telephone calls to Quitline: a semiparametric modelling approach. J. Epidemiol. Community Health 60:180–82 [Google Scholar]
  28. Ezzati M, Friedman AB, Kulkarni SC, Murray CJ. 28.  2008. The reversal of fortunes: trends in county mortality and cross-county mortality disparities in the United States. PLoS Med. 5 e66: [Google Scholar]
  29. Freud S. 29.  1974. The Standard Edition of the Complete Psychological Works of Sigmund Freud: 11 Great Britain: Hogarth [Google Scholar]
  30. Freund M, Campbell E, Paul C, Sakrouge R, Lecathelinais C. 30.  et al. 2009. Increasing hospital-wide delivery of smoking cessation care for nicotine-dependent in-patients: a multi-strategic intervention trial. Addiction 104:839–49 [Google Scholar]
  31. 31. Gambia Hepatitis Study Group 1987. The Gambia Hepatitis Intervention Study. Cancer Res. 47:5782–87 [Google Scholar]
  32. Glasziou P, Vandenbroucke J, Chalmers I. 32.  2004. Assessing the quality of research. BMJ 328:39–41 [Google Scholar]
  33. Green LW, Kreuter MW. 33.  1991. Health Promotion Planning: An Educational and Environmental Approach Mountain View, CA: Mayfield, 2nd ed.. [Google Scholar]
  34. Han PK, Coates RJ, Uhler RJ, Breen N. 34.  2006. Decision making in prostate-specific antigen screening: National Health Interview Survey, 2000. Am. J. Prev. Med. 30:394–404 [Google Scholar]
  35. Hancock L, Sanson-Fisher R, Perkins J, Corkrey R, Burton R, Reid S. 35.  2001. Effect of a community action intervention on cervical cancer screening rates in rural Australian towns: the CART Project. Prev. Med. 32:109–17 [Google Scholar]
  36. Hancock L, Sanson-Fisher R, Perkins J, Girgis A, Howley P, Schofield M. 36.  2001. The effect of a community action intervention on adolescent smoking rates in rural Australian towns: the CART project. Prev. Med. 32:332–40 [Google Scholar]
  37. Hancock L, Sanson-Fisher R, Perkins J, McClintock A, Howley P, Gibberd R. 37.  2001. Effect of a community action program on adult quit smoking rates in rural Australian towns: the CART Project. Prev. Med. 32:118–27 [Google Scholar]
  38. Hancock L, Sanson-Fisher R, Redman S, Burton R, Burton L. 38.  et al. 1996. Community action for cancer prevention: overview of the Cancer Action in Rural Towns (CART) project, Australia. Health Prom. Int. 11:277–90 [Google Scholar]
  39. Harris RP, Helfand M, Woolf SH, Lohr KN, Mulrow CD. 39.  et al. 2001. Current methods of the U.S. Preventive Services Task Force: a review of the process. Am. J. Prev. Med. 20:21–35 [Google Scholar]
  40. Hawkins N, Sanson-Fisher RW, Shakeshaft A, D'Este C, Green LW. 40.  2007. The multiple baseline design for evaluating population-based research. Am. J. Prev. Med. 33:162–68 [Google Scholar]
  41. Hayes RJ, Moulton LH. 41.  2009. Cluster Randomised Trials London: CRC Press [Google Scholar]
  42. Henstridge J, Homel R, Mackay P. 42.  1997. The Long-Term Effects of Random Breath Testing in Four Australian States: A Time Series Analysis CR 162 Canberra, Aust.: Fed. Off. Road Saf. [Google Scholar]
  43. Heywood A, Sanson-Fisher R, Ring I, Mudge P. 43.  1994. Risk prevalence and screening for cancer by general practitioners. Prev. Med. 23:152–59 [Google Scholar]
  44. Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D. 44.  et al. 2011. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ 343:d5928 [Google Scholar]
  45. Higgins JPT, Green S. 45.  2009. Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.2, updated September Chichester, UK: Wiley [Google Scholar]
  46. Hillestad R, Bigelow J, Bower A, Girosi F, Meili R. 46.  et al. 2005. Can electronic medical record systems transform health care? Potential health benefits, savings, and costs. Health Aff. 24:1103–17 [Google Scholar]
  47. Horrocks S, Anderson E, Salisbury C. 47.  2002. Systematic review of whether nurse practitioners working in primary care can provide equivalent care to doctors. BMJ 324:819–23 [Google Scholar]
  48. Hussey MA, Hughes JP. 48.  2007. Design and analysis of stepped wedge cluster randomized trials. Contemp. Clin. Trials 28:182–91 [Google Scholar]
  49. Ilic D, O'Connor D, Green S, Wilt TJ. 49.  2006. Screening for prostate cancer. Cochrane Database Syst. Rev. 1:CD004720 [Google Scholar]
  50. Janz NK, Becker MH. 50.  1984. The health belief model: a decade later. Health Educ. Behav. 11:1–47 [Google Scholar]
  51. Jeffreys M, Stevanovic V, Tobias M, Lewis C, Ellison-Loschmann L. 51.  et al. 2005. Ethnic inequalities in cancer survival in New Zealand: linkage study. Am. J. Public Health 95:834–37 [Google Scholar]
  52. Katsuta T, Saitoh A. 52.  2012. Haemophilus influenzae type b vaccine. J. Infect. Dis. 206:1645–51 [Google Scholar]
  53. Killam WP, Tambatamba BC, Chintu N, Rouse D, Stringer E. 53.  et al. 2010. Antiretroviral therapy in antenatal care to increase treatment initiation in HIV-infected pregnant women: a stepped-wedge evaluation. AIDS 24:85–91 [Google Scholar]
  54. Kitchen MS, Ransley JK, Greenwood DC, Clarke GP, Conner MT. 54.  et al. 2009. Study protocol: a cluster randomised controlled trial of a school based fruit and vegetable intervention—Project Tomato. BMC Health Serv. Res. 9:101 [Google Scholar]
  55. Klar N, Donner A. 55.  1997. The merits of matching in community intervention trials: a cautionary tale. Stat. Med. 16:1753–64 [Google Scholar]
  56. Kotz D, Spigt M, Arts IC, Crutzen R, Viechtbauer W. 56.  2012. Researchers should convince policy makers to perform a classic cluster randomized controlled trial instead of a stepped wedge design when an intervention is rolled out. J. Clin. Epidemiol. 65:1255–56 [Google Scholar]
  57. Kotz D, Spigt M, Arts IC, Crutzen R, Viechtbauer W. 57.  2012. Use of the stepped wedge design cannot be recommended: a critical appraisal and comparison with the classic cluster randomized controlled trial design. J. Clin. Epidemiol. 65:1249–52 [Google Scholar]
  58. Leung WC. 58.  2001. Balancing statistical and clinical significance in evaluating treatment effects. Postgrad. Med. J. 77:201–4 [Google Scholar]
  59. Macintyre S. 59.  2003. Evidence based policy making: Impact on health inequalities still needs to be assessed. BMJ 326:5–6 [Google Scholar]
  60. Marmot M. 60.  2005. Social determinants of health inequalities. Lancet 365:1099–104 [Google Scholar]
  61. Mdege ND, Man MS, Taylor (nee Brown) CA, Torgerson DJ. 61.  2012. There are some circumstances where the stepped-wedge cluster randomized trial is preferable to the alternative: no randomized trial at all. Response to the commentary by Kotz and colleagues. J. Clin. Epidemiol. 65:1253–54 [Google Scholar]
  62. Mdege ND, Man MS, Torgerson DJ. 62.  2011. Systematic review of stepped wedge cluster randomized trials shows that design is particularly used to evaluate interventions during routine implementation. J. Clin. Epidemiol. 64:936–48 [Google Scholar]
  63. Mercer S, DeVinney B, Fine L, Green L, Dougherty D. 63.  2007. Study designs for effectiveness and translation research: identifying trade-offs. Am. J. Prev. Med. 33:139–54 [Google Scholar]
  64. Michie S, van Stralen MM, West R. 64.  2011. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement. Sci. 6:42–53 [Google Scholar]
  65. Middleton S, Levi C, Ward J, Grimshaw J, Griffiths R. 65.  et al. 2009. Fever, hyperglycaemia and swallowing dysfunction management in acute stroke: a cluster randomised controlled trial of knowledge transfer. Implement. Sci. 4:16–27 [Google Scholar]
  66. Middleton S, McElduff P, Ward J, Grimshaw J, Dale S. 66.  et al. 2011. Implementation of evidence-based treatment protocols to manage fever, hyperglycaemia, and swallowing dysfunction in acute stroke (QASC): a cluster randomised controlled trial. Lancet 378:1699–706 [Google Scholar]
  67. Murray DM. 67.  1998. Design and Analysis of Group-Randomized Trials New York: Oxford Univ. Press [Google Scholar]
  68. Nestle M. 68.  2005. Preventing childhood diabetes: the need for public health intervention. Am. J. Public Health 95:1497–99 [Google Scholar]
  69. Ozanne-Smith J, Ashby K, Newstead S, Stathakis V, Clapperton A. 69.  2004. Firearm related deaths: the impact of regulatory reform. Inj. Prev. 10:280–86 [Google Scholar]
  70. Pan SY, Cameron C, DesMeules M, Morrison H, Craig CL, Jiang X. 70.  2009. Individual, social, environmental, and physical environmental correlates with physical activity among Canadians: a cross-sectional study. BMC Public Health 9:21–33 [Google Scholar]
  71. Pawson R, Tilley N. 71.  2006. Realist evaluation. DPRN Them. Meet. 2006 Rep. Eval. Amsterdam: Dev. Policy Rev. Netw. (DPRN) http://www.communitymatters.com.au/RE_chapter.pdf [Google Scholar]
  72. Petticrew M, Whitehead M, Macintyre S, Graham H, Egan M. 72.  2004. Evidence for public health policy on inequalities: 1: The reality according to policymakers. J. Epidemiol. Community Health 58:811–16 [Google Scholar]
  73. Reason P, Bradbury H. 73.  2008. The Sage Handbook of Action Research: Participative Inquiry and Practice Thousand Oaks, CA: Sage [Google Scholar]
  74. Reichardt CS, Mark MM. 74.  2004. Quasi-experimentation. Handbook of Practical Program Evaluation HH Wholey, JK Newcomer 126–49 San Francisco, CA: Jossey-Bass Inc. [Google Scholar]
  75. Reid AL, Webb GR, Hennrikus D, Fahey PP, Sanson-Fisher RW. 75.  1986. Detection of patients with high alcohol intake by general practitioners. BMJ Clin. Res. 293:735–37 [Google Scholar]
  76. Reid JL, Hammond D, Boudreau C, Fong GT, Siahpush M. 76.  2010. Socioeconomic disparities in quit intentions, quit attempts, and smoking abstinence among smokers in four western countries: findings from the International Tobacco Control Four Country Survey. Nicotine Tob. Res. 12:S20–33 [Google Scholar]
  77. Rogers CR, Skinner BF. 77.  1956. Some issues concerning the control of human behaviour. Science 124:1057–66 [Google Scholar]
  78. Rosenthal MB, Frank RG, Li Z, Epstein AM. 78.  2005. Early experience with pay-for-performance: from concept to practice. JAMA 294:1788–93 [Google Scholar]
  79. Rychetnik L, Frommer M, Hawe P, Shiell A. 79.  2002. Criteria for evaluating evidence on public health interventions. J. Epidemiol. Community Health 56:119–27 [Google Scholar]
  80. Sanson-Fisher R, Bonevski B, Green L, D'Este C. 80.  2007. Limitations of the randomised controlled trial in evaluating population-based health interventions. Am. J. Prev. Med. 33:155–61 [Google Scholar]
  81. Sanson-Fisher RW, Redman S, Oak S, Webb G. 81.  1988. The burden of illness that is imposed by drug abuse and the allocation of research monies in the field: three views. Med. J. Aust. 149:134–38 [Google Scholar]
  82. Saunders RP, Evans MH, Joshi P. 82.  2005. Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health Promot. Pract. 6:134–47 [Google Scholar]
  83. Schaap MM, Kunst AE, Leinsalu M, Regidor E, Ekholm O. 83.  et al. 2008. Effect of nationwide tobacco control policies on smoking cessation in high and low educated groups in 18 European countries. Tob. Control 17:248–55 [Google Scholar]
  84. Shadish WR, Cook TD, Campbell DT. 84.  2002. Experimental and Quasi-Experimental Designs for Generalized Causal Inference Boston/New York: Houghton Mifflin [Google Scholar]
  85. Shakeshaft A, Doran C, Petrie D, Breen C, Havard A. 85.  et al. 2012. The Alcohol Action in Rural Communities (AARC) Project: Detailed Description of the Interventions and Their Costs. Sydney: Natl. Drug Alcohol Res. Cent., Univ. N.S.W. [Google Scholar]
  86. Shakeshaft A, Doran C, Petrie D, Breen C, Havard A. 86.  et al. 2012. The Alcohol Action in Rural Communities (AARC) Project: Working with Communities to Select, Implement and Measure the Impact of Strategies to Reduce Alcohol-Related Harm. Deakin, Aust.: Found. Alcohol Res. Educ. [Google Scholar]
  87. Shen D, Lu Z. 87.  2006. Randomization in clinical trial studies Pap. Po06 presented at PharmaSUG, May 21–24, Bonita Springs, Fla. http://www.lexjansen.com/pharmasug/2006/posters/po06.pdf [Google Scholar]
  88. Swan J, Breen N, Coates RJ, Rimer BK, Lee NC. 88.  2003. Progress in cancer screening practices in the United States: results from the 2000 National Health Interview Survey. Cancer 97:1528–40 [Google Scholar]
  89. Tabrizi SN, Brotherton JM, Kaldor JM, Skinner SR, Cummins E. 89.  et al. 2012. Fall in human papillomavirus prevalence following a national vaccination program. J. Infect. Dis. 206:1645–51 [Google Scholar]
  90. Thompson B, Coronado G, Snipes SA, Puschel K. 90.  2003. Methodologic advances and ongoing challenges in designing community-based health promotion programs. Annu. Rev. Public Health 24:315–40 [Google Scholar]
  91. Tully J, Ninis N, Booy R, Viner R. 91.  2000. The new system of review by multicentre research ethics committees: prospective study. BMJ 320:1179–82 [Google Scholar]
  92. Volmink J, Garner P. 92.  2007. Directly observed therapy for treating tuberculosis. Cochrane Database Syst. Rev. (4):CD003343 [Google Scholar]
  93. Wakefield MA, Loken B, Hornik RC. 93.  2010. Use of mass media campaigns to change health behaviour. Lancet 376:1261–71 [Google Scholar]
  94. Wang J, Etter JF. 94.  2004. Administering an effective health intervention for smoking cessation online: the international users of Stop-Tabac. Prev. Med. 39:962–68 [Google Scholar]
  95. Wendy B. 95.  2012. Vaccination with 3-dose paediatric rotavirus vaccine (RotaTeq): impact on the timeliness of uptake of the primary course of DTPa vaccine. Vaccine 30:5293–97 [Google Scholar]
  96. 96. WHO (World Health Organ.) 2005. Preventing Chronic Diseases: A Vital Investment Geneva: WHO [Google Scholar]
  97. 97. WHO (World Health Organ.) 2011. Global Status Report on Noncommunicable Diseases 2010: Description of the Global Burden of NCDs, Their Risk Factors and Determinants Geneva: WHO [Google Scholar]
  98. 98. WHO Global Tuberc. Program 1997. WHO Report on the Tuberculosis Epidemic 1997. Geneva: WHO [Google Scholar]
  99. Wilbur J. 99.  2008. Prostate cancer screening: the continuing controversy. Am. Fam. Physician 78:1377–84 [Google Scholar]
  100. Wolf A, Wender RC, Etzioni RB, Thompson IM, D'Amico AV. 100.  et al. 2010. American Cancer Society guideline for the early detection of prostate cancer: update 2010. CA Cancer J. Clin. 60:70–98 [Google Scholar]
  101. Zaza S, Briss PA, Harris KW. 101.  2005. The Guide to Community Preventive Services: What Works to Promote Health? New York: Oxford Univ. Press [Google Scholar]
  102. Ziglio E. 102.  1997. How to move towards evidence-based health promotion interventions. Promot. Educ. 4:29–33 [Google Scholar]

Data & Media loading...

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error