Large-scale public policy changes are often recommended to improve public health. Despite varying widely—from tobacco taxes to poverty-relief programs—such policies present a common dilemma to public health researchers: how to evaluate their health effects when randomized controlled trials are not possible. Here, we review the state of knowledge and experience of public health researchers who rigorously evaluate the health consequences of large-scale public policy changes. We organize our discussion by detailing approaches to address three common challenges of conducting policy evaluations: distinguishing a policy effect from time trends in health outcomes or preexisting differences between policy-affected and -unaffected communities (using difference-in-differences approaches); constructing a comparison population when a policy affects a population for whom a well-matched comparator is not immediately available (using propensity score or synthetic control approaches); and addressing unobserved confounders by utilizing quasi-random variations in policy exposure (using regression discontinuity, instrumental variables, or near-far matching approaches).


Article metrics loading...

Loading full text...

Full text loading...


Literature Cited

  1. Abadie A, Diamond A, Hainmueller J. 1.  2010. Synthetic control methods for comparative case studies: estimating the effect of California's tobacco control program. J. Am. Stat. Assoc. 105:493–505 [Google Scholar]
  2. Angell SY, Silver LD, Goldstein GP, Johnson CM, Deitcher DR. 2.  et al. 2009. Cholesterol control beyond the clinic: New York City's trans fat restriction. Ann. Intern. Med. 151:2129–34 [Google Scholar]
  3. Angrist JD, Pischke J-S. 3.  2008. Mostly Harmless Econometrics: An Empiricist's Companion Princeton, NJ: Princeton Univ. Press
  4. Austin PC. 4.  2009. The relative ability of different propensity score methods to balance measured covariates between treated and untreated subjects in observational studies. Med. Decis. Making 29661–77 [Google Scholar]
  5. Austin PC. 5.  2009. Type I error rates, coverage of confidence intervals, and variance estimation in propensity-score matched analyses. Int. J. Biostat. 5:113 [Google Scholar]
  6. Austin PC. 6.  2011. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivar. Behav. Res. 46:3399–424 [Google Scholar]
  7. Austin PC, Grootendorst P, Anderson GM. 7.  2007. A comparison of the ability of different propensity score models to balance measured variables between treated and untreated subjects: a Monte Carlo study. Stat. Med. 26:4734–53 [Google Scholar]
  8. Austin PC, Mamdani MM. 8.  2006. A comparison of propensity score methods: a case-study estimating the effectiveness of post-AMI statin use. Stat. Med. 25:122084–106 [Google Scholar]
  9. Baiocchi M, Small DS, Lorch S, Rosenbaum PR. 9.  2010. Building a stronger instrument in an observational study of perinatal care for premature infants. J. Am. Stat. Assoc. 105:4921285–96 [Google Scholar]
  10. Baiocchi M, Small DS, Yang L, Polsky D, Groeneveld PW. 10.  2012. Near/far matching: a study design approach to instrumental variables. Health Serv. Outcomes Res. Methodol. 12:4237–53 [Google Scholar]
  11. Barry CL, Busch SH. 11.  2007. Do state parity laws reduce the financial burden on families of children with mental health care needs?. Health Serv. Res. 42:3 Pt. 11061–84 [Google Scholar]
  12. Basu S, Seligman HK, Gardner C, Bhattacharya J. 12.  2014. Ending SNAP subsidies for sugar-sweetened beverages could reduce obesity and type 2 diabetes. Health Aff 33:61032–39 [Google Scholar]
  13. Baum CF, Schaffer ME, Stillman S. 13.  2007. Enhanced routines for instrumental variables/GMM estimation and testing. Stata J 7:4465–506 [Google Scholar]
  14. Baum CL II. 14.  2010. The effects of Food Stamps on obesity Work. Pap., Dep. Econ. Financ., Middle Tenn. State Univ., Murfreesboro
  15. Bertrand M, Duflo E, Mullainathan S. 15.  2004. How much should we trust differences-in-differences estimates?. Q. J. Econ. 119:1249–75 [Google Scholar]
  16. Biswas R, Ray SK, Dobe M, Dasgupta S, Mandal A. 16.  2001. An overview of multicentric training workshops for public health professionals on reproductive and child health programme in India. Indian J. Public Health 46:378–85 [Google Scholar]
  17. Brown TT, Martinez-Gutierrez MS, Navab B. 17.  2014. The impact of changes in county public health expenditures on general health in the population. Health Econ. Policy Law 9:3251–69 [Google Scholar]
  18. Conley D, McCabe BJ. 18.  2012. Bribery or just desserts? Evidence on the influence of Congressional reproductive policy voting patterns on PAC contributions from exogenous variation in the sex mix of legislator offspring. Soc. Sci. Res. 41:1120–29 [Google Scholar]
  19. De Jaeger S. 19.  2009. Residual household waste: from pay-per-bag to pay-per-kilogram. An evaluation study for flanders. Waste Manag. Res. 28:330–39 [Google Scholar]
  20. De Neve J-W, Fink G, Subramanian SV, Moyo S, Bor J. 20.  2015. Length of secondary schooling and risk of HIV infection in Botswana: evidence from a natural experiment. Lancet Glob. Health 3:8e470–77 [Google Scholar]
  21. De Vogli R, Kouvonen A, Gimeno D, De Vogli R, Kouvonen A, Gimeno D. 21.  2014. The influence of market deregulation on fast food consumption and body mass index: a cross-national time series analysis. Bull. World Health Organ. 92:299–107A [Google Scholar]
  22. DeVoe JE, Marino M, Gold R, Hoopes MJ, Cowburn S. 22.  et al. 2015. Community health center use after Oregon's randomized Medicaid experiment. Ann. Fam. Med. 13:4312–20 [Google Scholar]
  23. Dimick JB, Ryan AM. 23.  2014. Methods for evaluating changes in health care policy: the difference-in-differences approach. JAMA 312:222401–2 [Google Scholar]
  24. Evans WN, Garthwaite CL. 24.  2014. Giving mom a break: the impact of higher EITC payments on maternal health. Am. Econ. J. Econ. Policy 6:2258–90 [Google Scholar]
  25. Fell JC, Scherer M, Thomas S, Voas RB. 25.  2014. Effectiveness of social host and fake identification laws on reducing underage drinking driver fatal crashes. Traffic Inj. Prev. 15:Suppl. 1S64–73 [Google Scholar]
  26. Fertig AR, Reingold DA. 26.  2007. Public housing, health, and health behaviors: Is there a connection?. J. Policy Anal. Manag. 26:4831–60 [Google Scholar]
  27. Friel S, Marmot MG. 27.  2011. Action on the social determinants of health and health inequities goes global. Annu. Rev. Public Health 32:225–36 [Google Scholar]
  28. Gantner LA, Olson CM. 28.  2012. Evaluation of public health professionals’ capacity to implement environmental changes supportive of healthy weight. Eval. Progr. Plan. 35:3407–16 [Google Scholar]
  29. Glantz SA. 29.  1993. Changes in cigarette consumption, prices, and tobacco industry revenues associated with California's proposition 99. Tob. Control 2:4311–14 [Google Scholar]
  30. Gormley WT Jr., Gayer T, Phillips D, Dawson B. 30.  2005. The effects of universal pre-K on cognitive development. Dev. Psychol. 41:6872–84 [Google Scholar]
  31. Gyorkos TW, Maheu-Giroux M, Blouin B, Creed-Kanashiro H, Casapía M. 31.  et al. 2012. A hospital policy change toward delayed cord clamping is effective in improving hemoglobin levels and anemia status of 8-month-old Peruvian infants. J. Trop. Pediatr. 58:6435–40 [Google Scholar]
  32. Halpern SD, French B, Small DS, Saulsgiver K, Harhay MO. 32.  et al. 2015. Randomized trial of four financial-incentive programs for smoking cessation. N. Engl. J. Med. 372:222108–17 [Google Scholar]
  33. Hanandita W, Tampubolon G. 33.  2014. Does poverty reduce mental health? An instrumental variable analysis. Soc. Sci. Med. 113:59–67 [Google Scholar]
  34. Hoynes H, Miller D, Simon D. 34.  2015. Income, the Earned Income Tax Credit, and infant health. Am. Econ. J. Econ. Policy 7:1172–211 [Google Scholar]
  35. Ikenwilo D. 35.  2013. A difference-in-differences analysis of the effect of free dental check-ups in Scotland. Soc. Sci. Med. 83:10–18 [Google Scholar]
  36. Imbens GW, Wooldridge J. 36.  2007. Difference-in-differences estimation Natl. Bur. Econ. Res. Work. Pap. http://www.nber.org/WNE/lect_10_diffindiffs.pdf
  37. Jayawardhana J, Welton JM, Lindrooth RC. 37.  2014. Is there a business case for magnet hospitals? Estimates of the cost and revenue implications of becoming a magnet. Med. Care 525400–6 [Google Scholar]
  38. Johar M. 38.  2009. The impact of the Indonesian health card program: a matching estimator approach. J. Health Econ. 28:135–53 [Google Scholar]
  39. Johnston DW, Lordan G, Shields MA, Suziedelyte A. 39.  2015. Education and health knowledge: evidence from UK compulsory schooling reform. Soc. Sci. Med. 127:92–100 [Google Scholar]
  40. Kadiyala S, Strumpf EC. 40.  2011. Are United States and Canadian cancer screening rates consistent with guideline information regarding the age of screening initiation?. Int. J. Qual. Health Care 23:6611–20 [Google Scholar]
  41. Kagohashi K, Tsurumi T, Managi S. 41.  2015. The effects of international trade on water use. PLOS ONE 10:7e0132133 [Google Scholar]
  42. Karaca-Mandic P, Choi-Yoo SJ, Lee J, Scal P. 42.  2014. Family out-of-pocket health care burden and children's unmet needs or delayed health care. Acad. Pediatr. 14:1101–8 [Google Scholar]
  43. King G, Tanner MA, Rosen O. 43.  2004. Ecological Inference: New Methodological Strategies. New York: Cambridge Univ. Press
  44. Konetzka RT, Yi D, Norton EC, Kilpatrick KE. 44.  2004. Effects of Medicare payment changes on nursing home staffing and deficiencies. Health Serv. Res. 39:3463–88 [Google Scholar]
  45. Kozhimannil KB, Attanasio LB, McGovern PM, Gjerdingen DK, Johnson PJ. 45.  2013. Reevaluating the relationship between prenatal employment and birth outcomes: a policy-relevant application of propensity score matching. Womens Health Issues 23:2e77–85 [Google Scholar]
  46. Lambon-Quayefio MP, Owoo NS. 46.  2014. Examining the influence of antenatal care visits and skilled delivery on neonatal deaths in Ghana. Appl. Health Econ. Health Policy 12:5511–22 [Google Scholar]
  47. Lee DS, Lemieux T. 47.  2009. Regression discontinuity designs in economics. J. Econ. Lit. 48:281–355 [Google Scholar]
  48. Leung CW, Ding EL, Catalano PJ, Villamor E, Rimm EB, Willett WC. 48.  2012. Dietary intake and dietary quality of low-income adults in the Supplemental Nutrition Assistance Program. Am. J. Clin. Nutr. 96:5977–88 [Google Scholar]
  49. Lillico HG, Hanning R, Findlay S, Hammond D. 49.  2015. The effects of calorie labels on those at high-risk of eating pathologies: a pre-post intervention study in a university cafeteria. Public Health 129:6732–39 [Google Scholar]
  50. Liu H, Han X, Xiao Q, Li S, Feldman MW. 50.  2015. Family structure and quality of life of elders in rural china: the role of the new rural social pension. J. Aging Soc. Policy 27:2123–38 [Google Scholar]
  51. Lorch SA, Baiocchi M, Ahlberg CE, Small DS. 51.  2012. The differential impact of delivery hospital on the outcomes of premature infants. Pediatrics 130:2270–78 [Google Scholar]
  52. McCormick D, Hanchate AD, Lasser KE, Manze MG, Lin M. 52.  et al. 2015. Effect of Massachusetts healthcare reform on racial and ethnic disparities in admissions to hospital for ambulatory care sensitive conditions: retrospective analysis of hospital episode statistics. BMJ 350:h1480 [Google Scholar]
  53. McCrary J. 53.  2008. Manipulation of the running variable in the regression discontinuity design: a density test. J. Econom. 142:2698–714 [Google Scholar]
  54. Meara E, Kotagal UR, Atherton HD, Lieu TA. 54.  2004. Impact of early newborn discharge legislation and early follow-up visits on infant outcomes in a state Medicaid population. Pediatrics 113:61619–27 [Google Scholar]
  55. Meret-Hanke LA. 55.  2011. Effects of the Program of All-inclusive Care for the Elderly on hospital use. Gerontologist 51:6774–85 [Google Scholar]
  56. Mukamel DB, Kang T, Collier E, Harrington C. 56.  2012. The relationship of California's Medicaid reimbursement system to nurse staffing levels. Med. Care 5010836–42 [Google Scholar]
  57. Mulvale G, Hurley J. 57.  2008. Insurance coverage and the treatment of mental illness: effect on medication and provider use. J. Ment. Health Policy Econ. 11:4177–99 [Google Scholar]
  58. Nanda P. 58.  1999. Women's participation in rural credit programmes in Bangladesh and their demand for formal health care: Is there a positive impact?. Health Econ 8:5415–28 [Google Scholar]
  59. Neuman MD, Rosenbaum PR, Ludwig JM, Zubizarreta JR, Silber JH. 59.  2014. Anesthesia technique, mortality, and length of stay after hip fracture surgery. JAMA 311:242508–17 [Google Scholar]
  60. Nishi A, McWilliams JM, Noguchi H, Hashimoto H, Tamiya N, Kawachi I. 60.  2012. Health benefits of reduced patient cost sharing in Japan. Bull. World Health Organ. 90:6426–35a [Google Scholar]
  61. Norwood P, Eberth B, Farrar S, Anable J, Ludbrook A. 61.  2014. Active travel intervention and physical activity behaviour: an evaluation. Soc. Sci. Med. 113:50–58 [Google Scholar]
  62. Olsho LE, Klerman JA, Ritchie L, Wakimoto P, Webb KL, Bartlett S. 62.  2015. Increasing child fruit and vegetable intake: findings from the US Department of Agriculture fresh fruit and vegetable program. J. Acad. Nutr. Diet. 115:1283–90 [Google Scholar]
  63. Patel MS, Volpp KG, Small DS, Hill AS, Even-Shoshan O. 63.  et al. 2014. Association of the 2011 ACGME resident duty hour reforms with mortality and readmissions among hospitalized medicare patients. JAMA 312:222364–73 [Google Scholar]
  64. Pomeranz JL. 64.  2015. Implications of the Supplemental Nutrition Assistance Program tax exemption on sugar-sweetened beverage taxes. Am. J. Public Health 105:112191–93 [Google Scholar]
  65. Porter D. 65.  1994. The History of Public Health and the Modern State Atlanta: Rodopi
  66. Potylycki MJ, Kimmel SR, Ritter M, Capuano T, Gross L. 66.  et al. 2006. Nonpunitive medication error reporting: 3-year findings from one hospital's primum non nocere initiative. J. Nurs. Adm. 36:7–8370–76 [Google Scholar]
  67. Rajaram R, Chung JW, Jones AT, Cohen ME, Dahlke AR. 67.  et al. 2014. Association of the 2011 ACGME resident duty hour reform with general surgery patient outcomes and with resident examination performance. JAMA 312:222374–84 [Google Scholar]
  68. Remler DK, Van Ryzin GG. 68.  2010. Natural and quasi experiments. Research Methods in Practice: Strategies for Description and Causation427–64 Thousand Oaks, CA: SAGE Publ. [Google Scholar]
  69. Rosenbaum PR. 69.  1987. The role of a second control group in an observational study. Stat. Sci. 2:3292–306 [Google Scholar]
  70. Salas M, Hotman A, Stricker BH. 70.  1999. Confounding by indication: an example of variation in the use of epidemiologic terminology. Am. J. Epidemiol. 149:11981–83 [Google Scholar]
  71. Salehi-Isfahani D, Abbasi-Shavazi MJ, Hosseini-Chavoshi M. 71.  2010. Family planning and fertility decline in rural Iran: the impact of rural health clinics. Health Econ 19:Suppl.159–80 [Google Scholar]
  72. Sassi F, Belloni A, Capobianco C, Alemanno A. 72.  2014. Taxation and economic incentives on health-related commodities: alcohol, tobacco and food. Regulating Lifestyle—Europe, Alcohol, Tobacco and Unhealthy Diets A Alemanno, A Garde Cambridge, UK: Cambridge Univ. Press [Google Scholar]
  73. Shen Y-C, Wu VY. 73.  2013. Reductions in Medicare payments and patient outcomes: an analysis of 5 leading Medicare conditions. Med. Care 5111970–77 [Google Scholar]
  74. Slawson DL, Southerland J, Lowe EF, Dalton WT, Pfortmiller DT, Schetzina K. 74.  2013. Go Slow Whoa meal patterns: cafeteria staff and teacher perceptions of effectiveness in winning with wellness schools. J. Sch. Health 83:7485–92 [Google Scholar]
  75. Sommers BD, Baicker K, Epstein AM. 75.  2012. Mortality and access to care among adults after state Medicaid expansions. N. Engl. J. Med. 367:111025–34 [Google Scholar]
  76. Stock JH, Yogo M. 76.  2005. Testing for weak instruments in linear IV regression. Identification and Inference for Econometric Models DWK Andrews 80–108 New York: Cambridge Univ. Press [Google Scholar]
  77. Subramanian SV, Jones K, Kaddour A, Krieger N. 77.  2009. Revisiting Robinson: the perils of individualistic and ecologic fallacy. Int. J. Epidemiol. 38:2342–60 [Google Scholar]
  78. Tsai AC, Venkataramani AS. 78.  2015. The causal effect of education on HIV stigma in Uganda: evidence from a natural experiment. Soc. Sci. Med. 142:37–46 [Google Scholar]
  79. 79. WHO (World Health Organ.) Comm. Soc. Determ. Health. 2008. Closing a Gap in a Generation: Health Equity Through Action on the Social Determinants of Health—Final Report Geneva: WHO http://apps.who.int/iris/bitstream/10665/43943/1/9789241563703_eng.pdf
  80. Wooldridge J. 80.  2008. Introductory Econometrics: A Modern Approach Mason, OH: South-Western, Cengage Learn.

Data & Media loading...

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error