1932

Abstract

The difference in difference (DID) design is a quasi-experimental research design that researchers often use to study causal relationships in public health settings where randomized controlled trials (RCTs) are infeasible or unethical. However, causal inference poses many challenges in DID designs. In this article, we review key features of DID designs with an emphasis on public health policy research. Contemporary researchers should take an active approach to the design of DID studies, seeking to construct comparison groups, sensitivity analyses, and robustness checks that help validate the method's assumptions. We explain the key assumptions of the design and discuss analytic tactics, supplementary analysis, and approaches to statistical inference that are often important in applied research. The DID design is not a perfect substitute for randomized experiments, but it often represents a feasible way to learn about casual relationships. We conclude by noting that combining elements from multiple quasi-experimental techniques may be important in the next wave of innovations to the DID approach.

Loading

Article metrics loading...

/content/journals/10.1146/annurev-publhealth-040617-013507
2018-04-01
2024-03-28
Loading full text...

Full text loading...

/deliver/fulltext/publhealth/39/1/annurev-publhealth-040617-013507.html?itemId=/content/journals/10.1146/annurev-publhealth-040617-013507&mimeType=html&fmt=ahah

Literature Cited

  1. Abadie A, Diamond A, Hainmueller J. 1.  2010. Synthetic control methods for comparative case studies: estimating the effect of California's tobacco control program. J. Am. Stat. Assoc. 105:490493–505 [Google Scholar]
  2. Almond D, Edlund L, Palme M. 2.  2009. Chernobyl's subclinical legacy: prenatal exposure to radioactive fallout and school outcomes in Sweden. Q. J. Econ. 124:41729–72 [Google Scholar]
  3. Alpert A. 3.  2016. The anticipatory effects of Medicare Part D on drug utilization. J. Health Econ. 49:28–45 [Google Scholar]
  4. Anderson DM, Hansen B, Rees DI. 4.  2015. Medical marijuana laws and teen marijuana use. Am. Law Econ. Rev. 17:2495–528 [Google Scholar]
  5. Anderson DM, Rees DI, Sabia JJ. 5.  2014. Medical marijuana laws and suicides by gender and age. Am. J. Public Health 104:122369–76 [Google Scholar]
  6. Angrist JD, Pischke JS. 6.  2008. Mostly Harmless Econometrics: An Empiricist's Companion Princeton, NJ: Princeton Univ. Press
  7. Angrist J, Rokkanen M. 7.  2012. Wanna get away? RD identification away from the cutoff NBER Work. Pap. 18662 Cambridge, MA:
  8. Antwi YA, Moriya AS, Simon K. 8.  2013. Effects of federal policy to insure young adults: evidence from the 2010 Affordable Care Act's dependent-coverage mandate. Am. Econ. J. Econ. Policy 5:41–28 [Google Scholar]
  9. Atanasov V, Black B. 9.  2016. Shock-based causal inference in corporate finance and accounting research. Crit. Finance Rev. 5:207–304 [Google Scholar]
  10. Athey A, Imbens GW. 10.  2017. The state of applied econometrics: causality and policy evaluations. J. Econ. Perspect. 31:23–32 [Google Scholar]
  11. Bachhuber MA, Saloner B, Cunningham CO, Barry CL. 11.  2014. Medical cannabis laws and opioid analgesic overdose mortality in the United States, 1999–2010. JAMA Intern. Med. 174:101668–73 [Google Scholar]
  12. Baicker K, Taubman SL, Allen HL, Bernstein M, Gruber JH, Newhouse JP. 12.  et al. 2013. The Oregon experiment—effects of Medicaid on clinical outcomes. N. Engl. J. Med. 368:1713–22 [Google Scholar]
  13. Basu S, Meghani A, Siddiqi A. 13.  2017. Evaluating the health impact of large-scale public policy changes: classical and novel approaches. Annu. Rev. Public Health 38:351–70 [Google Scholar]
  14. Bedard K, Kuhn P. 14.  2015. Micro-marketing healthier choices: effects of personalized ordering suggestions on restaurant purchases. J. Health Econ. 39:106–22 [Google Scholar]
  15. Bell RM, McCaffrey DF. 15.  2002. Bias reduction in standard errors for linear regression with multi-stage samples. Surv. Methodol. 28:2169–81 [Google Scholar]
  16. Bellou A, Bhatt R. 16.  2013. Reducing underage alcohol and tobacco use: evidence from the introduction of vertical identification cards. J. Health Econ. 32:2353–66 [Google Scholar]
  17. Bertanha M, Imbens GW. 17.  2014. External validity in fuzzy regression discontinuity designs NBER Work. Pap. 20773 Cambridge, MA:
  18. Bertrand M, Duflo E, Mullainathan S. 18.  2004. How much should we trust differences-in-differences estimates?. Q. J. Econ. 119:1249–75 [Google Scholar]
  19. Besley T, Case A. 19.  2000. Unnatural experiments? Estimating the incidence of endogenous policies. Econ. J. 110:467672–94 [Google Scholar]
  20. Bleakley H. 20.  2007. Disease and development: evidence from hookworm eradication in the American South. Q. J. Econ. 122:173–117 [Google Scholar]
  21. Bitler MP, Carpenter CS. 21.  2016. Health insurance mandates, mammography, and breast cancer diagnoses. Am. Econ. J.: Econ. Policy 8:339–68 [Google Scholar]
  22. Bradford AC, Bradford WD. 22.  2016. Medical marijuana laws reduce prescription medication use in Medicare Part D. Health Aff 35:71230–36 [Google Scholar]
  23. Bramson H, Jarlais DCD, Arasteh K, Nugent A, Guardino V, Feelemyer J. 23.  et al. 2015. State laws, syringe exchange, and HIV among persons who inject drugs in the United States: history and effectiveness. J. Public Health Policy 36:2212–30 [Google Scholar]
  24. Brot-Goldberg ZC, Chandra A, Handel BR, Kolstad JT. 24.  2017. What does a deductible do? The impact of cost-sharing on health care prices, quantities, and spending dynamics. Q. J. Econ. 132:31261–318 [Google Scholar]
  25. Bullinger LR. 25.  2017. The effect of minimum wages on adolescent fertility: a nationwide analysis. Am. J. Public Health 107:3447–52 [Google Scholar]
  26. Burlig F, Preonas L, Woerman M. 26.  2017. Panel data and experimental design Work. Pap. 277a Energy Inst. Haas, Univ. Calif Berkeley: https://ei.haas.berkeley.edu/research/papers/WP277Appendix.pdf
  27. Bustamante AV, Mendez CA. 27.  2014. Health care privatization in Latin America: comparing divergent privatization approaches in Chile, Colombia, and Mexico. J. Health Polit. Policy Law 39:4841–86 [Google Scholar]
  28. Cameron AC, Gelbach JB, Miller DL. 28.  2008. Bootstrap-based improvements for inference with clustered errors. Rev. Econ. Stat. 90:3414–27 [Google Scholar]
  29. Cameron AC, Gelbach JB, Miller DL. 29.  2011. Robust inference with multiway clustering. J. Bus. Econ. Stat. 29:2238–49 [Google Scholar]
  30. Cameron AC, Miller DL. 30.  2015. A practitioner's guide to cluster-robust inference. J. Hum. Resour. 50:2317–72 [Google Scholar]
  31. Cantor J, Torres A, Abrams C, Elbel B. 31.  2015. Five years later: awareness of New York City's calorie labels declined, with no changes in calories purchased. Health Aff 34:111893–900 [Google Scholar]
  32. Card D, Shore-Sheppard L. 32.  2004. Using discontinuous eligibility rules to identify the effects of the federal Medicaid expansion on low-income children. Rev. Econ. Stat. 86:752–66 [Google Scholar]
  33. Carpenter C, Cook PJ. 33.  2008. Cigarette taxes and youth smoking: new evidence from national, state, and local Youth Risk Behavior Surveys. J. Health Econ. 27:2287–99 [Google Scholar]
  34. Carpenter C, Dobkin C. 34.  2017. The minimum legal drinking age and morbidity in the United States. Rev. Econ. Stat. 99:195–104 [Google Scholar]
  35. Cawley J, Frisvold DE. 35.  2017. The pass-through of taxes on sugar-sweetened beverages to retail prices: the case of Berkeley, California. J. Policy Anal. Manag. 36:2303–26 [Google Scholar]
  36. Chatterjee N, Shi J, García-Closas M. 36.  2016. Developing and evaluating polygenic risk prediction models for stratified disease prevention. Nat. Rev. Genet. 17:392–406 [Google Scholar]
  37. Cohodes SR, Grossman DS, Kleiner SA, Lovenheim MF. 37.  2016. The effect of child health insurance access on schooling: evidence from public insurance expansions. J. Hum. Resour. 51:3727–59 [Google Scholar]
  38. Conley TG, Taber CR. 38.  2011. Inference with “difference in differences” with a small number of policy changes. Rev. Econ. Stat. 93:1113–25 [Google Scholar]
  39. Cook PJ, Durrance CP. 39.  2013. The virtuous tax: lifesaving and crime-prevention effects of the 1991 federal alcohol-tax increase. J. Health Econ. 32:261–67 [Google Scholar]
  40. Cotti C, Tefft N. 40.  2013. Fast food prices, obesity, and the minimum wage. Econ. Hum. Biol. 11:2134–47 [Google Scholar]
  41. Craig P, Katikireddi SV, Leyland A, Popham F. 41.  2017. Natural experiments: an overview of methods, approaches, and contributions to public health intervention research. Annu. Rev. Public Health 38:39–56 [Google Scholar]
  42. Currie J, Gruber J. 42.  1996. Health insurance eligibility, utilization of medical care, and child health. Q. J. Econ. 111:2431–66 [Google Scholar]
  43. Currie J, Gruber J, Fischer M. 43.  1995. Physician payments and infant mortality: evidence from Medicaid fee policy. Am. Econ. Rev. 85:2106–11 [Google Scholar]
  44. Decker S. 44.  2009. Changes in Medicaid physician fees and patterns of ambulatory care. INQUIRY: J. Health Care Org. Provis. Financ. 46:3291–304 [Google Scholar]
  45. Dimick JB, Ryan AM. 45.  2014. Methods for evaluating changes in health care policy: the difference-in-differences approach. JAMA 312:222401–2 [Google Scholar]
  46. Donald SG, Lang K. 46.  2007. Inference with difference-in-differences and other panel data. Rev. Econ. Stat. 89:2221–33 [Google Scholar]
  47. Doudchenko N, Imbens GW. 47.  2016. Balancing, regression, difference-in-differences and synthetic control methods: a synthesis NBER Work. Pap. 22791 Cambridge, MA:
  48. Finkelstein A. 48.  2007. The aggregate effects of health insurance: evidence from the introduction of Medicare. Q. J. Econ. 122:11–37 [Google Scholar]
  49. Garthwaite C, Gross T, Notowidigdo MJ. 49.  2014. Public health insurance, labor supply, and employment lock. Q. J. Econ. 129:2653–96 [Google Scholar]
  50. Gebel M, Vossemer J. 50.  2014. The impact of employment transitions on health in Germany: a difference-in-differences propensity score matching approach. Soc. Sci. Med. 108:128–36 [Google Scholar]
  51. Goodman P, Agnew M, McCaffrey M, Paul G, Clancy L. 51.  2007. Effects of the Irish smoking ban on respiratory health of bar workers and air quality in Dublin pubs. Am. J. Respir. Crit. Care Med. 175:8840–45 [Google Scholar]
  52. Gruber J. 52.  1994. The incidence of mandated maternity benefits. Am. Econ. Rev. 84:3622–41 [Google Scholar]
  53. Gruber J, Simon K. 53.  2008. Crowd-out 10 years later: Have recent public insurance expansions crowded out private health insurance?. J. Health Econ. 27:2201–17 [Google Scholar]
  54. Hackmann M, Kolstadt J, Kowalski A. 54.  2015. Adverse selection and an individual mandate: when theory meets practice. Am. Econ. Rev. 105:31030–66 [Google Scholar]
  55. Hamersma S, Kim M. 55.  2013. Participation and crowd out: assessing the effects of parental Medicaid expansions. J. Health Econ. 32:1160–71 [Google Scholar]
  56. Hansen B, Sabia JJ, Rees DI. 56.  2017. Have cigarette taxes lost their bite? New estimates of the relationship between cigarette taxes and youth smoking. Am. J. Health Econ. 3:160–75 [Google Scholar]
  57. Harper S, Strumpf EC, Kaufman JS. 57.  2012. Do medical marijuana laws increase marijuana use? Replication study and extension. Ann. Epidemiol. 22:3207–12 [Google Scholar]
  58. Heim B, Lin L. 58.  2016. Does health reform lead to an increase in early retirement? Evidence from Massachusetts. ILR Rev 70:3704–32 [Google Scholar]
  59. Ho DE, Imai K, King G, Stuart EA. 59.  2007. Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Polit. Anal. 15:3199–236 [Google Scholar]
  60. Holahan J, Weil A, Wiener JM. 60.  2003. Federalism and Health Policy Washington, DC: Urban Inst. Press
  61. Hoynes H, Schanzenbach DW, Almond D. 61.  2016. Long-run impacts of childhood access to the safety net. Am. Econ. Rev. 106:4903–34 [Google Scholar]
  62. Imbens GW, Kolesar M. 62.  2016. Robust standard errors in small samples: some practical advice. Rev. Econ. Stat. 98:4701–12 [Google Scholar]
  63. Imbens GW, Rubin DB. 63.  2015. Causal Inference in Statistics, Social, and Biomedical Sciences Cambridge, UK: Cambridge Univ. Press
  64. Kaestner R, Garret B, Chen J, Gangopadhyaya A, Fleming C. 64.  2017. Effects of ACA Medicaid expansion on health insurance coverage and labor supply. J. Policy Anal. Manag. 36:3608–42 [Google Scholar]
  65. Ketcham JD, Simon KI. 65.  2008. Medicare Part D's effects on elderly patients’ drug costs and utilization. Am. J. Manag. Care 14:1114–22 [Google Scholar]
  66. Kolstad JT, Kowalski AE. 66.  2012. The impact of health care reform on hospital and preventive care: evidence from Massachusetts. J. Public Econ. 96:11–12909–29 [Google Scholar]
  67. Kowalski A. 67.  2016. Doing more when you're running LATE: applying marginal treatment effect methods to examine treatment effect heterogeneity in experiments NBER Work. Pap. 22363 Cambridge, MA:
  68. Kreif N, Grieve R, Hangartner D, Turner AJ, Nikolova S, Sutton M. 68.  2016. Examination of the synthetic control method for evaluating health policies with multiple treated units. Health Econ 25:121514–28 [Google Scholar]
  69. Kuehnle D, Wunder C. 69.  2016. The effects of smoking bans on self-assessed health: evidence from Germany. Health Econ 26:3321–37 [Google Scholar]
  70. Lane S, Hennes E, West T. 70.  2016. “I've got the power”: how anyone can do a power analysis of any type of study using simulation Presented at Soc. Personal. Soc. Psychol. Annu. Conv San Diego: http://meeting.spsp.org/2016/sites/default/files/Lane%2C%20Hennes%2C%20West%20SPSP%20Power%20Workshop%202016.pdf
  71. Lemos S. 71.  2005. Political variables as instruments for the minimum wage. B. E. J. Econ. Anal. Policy 4:11–33 [Google Scholar]
  72. Liang KY, Zeger SL. 72.  1986. Longitudinal data analysis using generalized linear models. Biometrika 73:113–22 [Google Scholar]
  73. Madrian BC. 73.  1994. Employment-based health insurance and job mobility: Is there evidence of job-lock?. Q. J. Econ. 109:127–54 [Google Scholar]
  74. Manning WG, Newhouse JP, Duan N, Keeler EB, Leibowitz A. 74.  1987. Health insurance and the demand for medical care: evidence from a randomized experiment. Am. Econ. Rev. 77:3251–77 [Google Scholar]
  75. Marcus J, Siedler T. 75.  2015. Reducing binge drinking? The effect of a ban on late-night off-premise alcohol sales on alcohol-related hospital stays in Germany. J. Public Econ. 123:55–77 [Google Scholar]
  76. Marks MS. 76.  2011. Minimum wages, employer-provided health insurance, and the non-discrimination law. Ind. Relat. J. Econ. Soc. 50:2241–62 [Google Scholar]
  77. McClelland G, Gault S. 77.  2017. The synthetic control method as a tool to understand state policy Res. Rep., Urban Inst Washington, DC:
  78. Miller S. 78.  2012. The effect of insurance on emergency room visits: an analysis of the 2006 Massachusetts health reform. J. Public Econ. 96:11–12893–908 [Google Scholar]
  79. Moulton BR. 79.  1990. An illustration of a pitfall in estimating the effects of aggregate variables on micro units. Rev. Econ. Stat. 72:2334–38 [Google Scholar]
  80. Muthén LK, Muthén BO. 80.  2009. How to use a Monte Carlo study to decide on sample size and determine power. Struct. Equ. Model.: Multidiscip. J. 9:4599–620 [Google Scholar]
  81. 81. NAMSDL (Natl. Alliance Model State Drug Laws). 2017. Use of marijuana for medicinal purposes: map of state laws NAMSDL, Manchester, IA. http://www.namsdl.org/library/CDC6A46B-F8E2-5F41-DBF6FBE02C0BC877/
  82. Nathan RP. 82.  2005. Federalism and health policy. Health Aff 24:61458–66 [Google Scholar]
  83. Obermeyer Z, Makar M, Abujaber S. 83.  2014. Association between the Medicare hospice benefit and health care utilization and costs for patients with poor-prognosis cancer. JAMA 312:181888–96 [Google Scholar]
  84. Paik M, Black B, Hyman D. 84.  2013. The receding tide of medical malpractice litigation part 2: effect of damage caps. J. Empir. Legal Stud. 10:639–69 [Google Scholar]
  85. Paik M, Black B, Hyman DA. 85.  2016. Damage caps and the labor supply of physicians: evidence from the third reform wave. Am. Law Econ. Rev. 18:2463–505 [Google Scholar]
  86. Pimentel SD, Kelz RR, Silber JH, Rosenbaum PR. 86.  2015. Large, sparse optimal matching with refined covariate balance in an observational study of the health outcomes produced by new surgeons. J. Am. Stat. Assoc. 110:510515–27 [Google Scholar]
  87. Pustejovsky JE, Tipton E. 87.  2016. Small sample methods for cluster-robust variance estimation and hypothesis testing in fixed effects models. J. Bus. Econ. Stat. In press. https://doi.org/10.1080/07350015.2016.1247004 [Crossref]
  88. Raifman J, Moscoe E, Austin B, McConnell M. 88.  2017. Difference-in-differences analysis of the association between state same-sex marriage policies and adolescent suicide attempts. JAMA Pediatr 171:4350–56 [Google Scholar]
  89. Rieger M, Wagner N, Bedi AS. 89.  2017. Universal health coverage at the macro level: synthetic control evidence from Thailand. Soc. Sci. Med. 172:46–55 [Google Scholar]
  90. Rosenbaum PR. 90.  2002. Covariance adjustment in randomized experiments and observational studies. Stat. Sci. 17:3286–327 [Google Scholar]
  91. Saloner B, Feder KA, Krawczyk N. 91.  2017. Closing the medication-assisted treatment gap for youth with opioid use disorder. JAMA Pediatr 171:8729–31 [Google Scholar]
  92. Shadish WR, Cook TD, Campbell DT. 92.  2002. Experimental and Quasi-Experimental Designs for Generalized Causal Inference Boston, MA: Houghton Mifflin Co.
  93. Simon D. 93.  2016. Does early life exposure to cigarette smoke permanently harm childhood welfare? Evidence from cigarette tax hikes. Am. Econ. J. Appl. Econ. 8:4128–59 [Google Scholar]
  94. Simon KI. 94.  2005. Adverse selection in health insurance markets? Evidence from state small-group health insurance reforms. J. Public Econ. 89:9–101865–77 [Google Scholar]
  95. Simon KI, Kaestner R. 95.  2004. Do minimum wages affect non-wage job attributes? Evidence on fringe benefits. ILR Rev 58:152–70 [Google Scholar]
  96. Simon K, Soni A, Cawley J. 96.  2017. The impact of health insurance on preventive care and health behaviors: evidence from the first two years of the ACA Medicaid expansions. J. Policy Anal. Manag. 36:2390–417 [Google Scholar]
  97. Snow J. 97.  1855. On the Mode of Communication of Cholera London: John Churchill
  98. Solon G, Haider SJ, Wooldridge JM. 98.  2015. What are we weighting for?. J. Hum. Resour. 50:2301–16 [Google Scholar]
  99. Somers MA, Zhu P, Jacob R, Bloom H. 99.  2013. The validity and precision of the comparative interrupted time series design and the difference-in-difference design in educational evaluation Work. Pap. Res. Methodol MDRC, New York: https://www.mdrc.org/sites/default/files/validity_precision_comparative_interrupted_time_series_design.pdf
  100. Sommers BD, Gunja MZ, Finegold K, Musco T. 100.  2015. Changes in self-reported insurance coverage, access to care, and health under the Affordable Care Act. JAMA 314:4366–74 [Google Scholar]
  101. Sommers BD, Kronick R. 101.  2012. The Affordable Care Act and insurance coverage for young adults. JAMA 307:9913–14 [Google Scholar]
  102. Trogdon JG, Shafer PR, Shah PD, Calo WA. 102.  2016. Are state laws granting pharmacists authority to vaccinate associated with HPV vaccination rates among adolescents?. Vaccine 34:384514–19 [Google Scholar]
  103. Wehby G, Dave D, Kaestner R. 103.  2016. Effects of the minimum wage on infant health NBER Work. Pap. 22373 Cambridge, MA:
  104. Wing C, Cook TD. 104.  2013. Strengthening the regression discontinuity design using additional design elements: a within-study comparison. J. Policy Anal. Manag. 32:4853–77 [Google Scholar]
  105. Wolfe B, Scrivner S. 105.  2005. The devil may be in the details: how the characteristics of SCHIP programs affect take-up. J. Policy Anal. Manag. 24:3499–522 [Google Scholar]
  106. Xiao L, Jiang Y, Liu X, Li Y, Gan Q, Liu F. 106.  2017. Smoking reduced in urban restaurants: the effect of Beijing Smoking Control Regulation. Tob. Control 26:e75–78 [Google Scholar]
/content/journals/10.1146/annurev-publhealth-040617-013507
Loading
  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error