1932
0
  1. Home
  2. A-Z Publications
  3. Annual Review of Public Health
  4. Volume 45, 2024
  5. Article

Annual Review of Public Health

Volume 45, 2024

Designing Difference-in-Difference Studies with Staggered Treatment Adoption: Key Concepts and Practical Guidelines

Abstract

Difference-in-difference (DID) estimators are a valuable method for identifying causal effects in the public health researcher's toolkit. A growing methods literature points out potential problems with DID estimators when treatment is staggered in adoption and varies with time. Despite this, no practical guide exists for addressing these new critiques in public health research. We illustrate these new DID concepts with step-by-step examples, code, and a checklist. We draw insights by comparing the simple 2 × 2 DID design (single treatment group, single control group, two time periods) with more complex cases: additional treated groups, additional time periods of treatment, and treatment effects possibly varying over time. We outline newly uncovered threats to causal interpretation of DID estimates and the solutions the literature has proposed, relying on a decomposition that shows how the more complex DIDs are an average of simpler 2 × 2 DID subexperiments.

Keyword(s): causal inference, difference-in-differences, natural experiments
Loading

Article metrics loading...

/content/journals/10.1146/annurev-publhealth-061022-050825
2024-05-20
2024-05-28
Loading full text...

Full text loading...

/deliver/fulltext/publhealth/45/1/annurev-publhealth-061022-050825.html?itemId=/content/journals/10.1146/annurev-publhealth-061022-050825&mimeType=html&fmt=ahah

Literature Cited

  1. 1.
    Athey S, Imbens GW. 2022.. Design-based analysis in Difference-In-Differences settings with staggered adoption. . J. Econom. 226:(1):62–79
     [Crossref] [Google Scholar]
  2. 2.
    Baker AC, Larcker DF, Wang CCY. 2022.. How much should we trust staggered difference-in-differences estimates?. J. Financ. Econ. 144:(2):370–95
     [Crossref] [Google Scholar]
  3. 3.
    Borusyak K, Jaravel X, Spiess J. 2022.. Revisiting event study designs: robust and efficient estimation. . arXiv:2108.12419v2 [econ.EM]
  4. 4.
    Caetano C, Callaway B, Payne S, Rodrigues HS. 2022.. Difference in differences with time-varying covariates. . arXiv:2202.02903 [econ]
  5. 5.
    Callaway B, Sant'Anna PHC. 2021.. Difference-in-Differences with multiple time periods. . J. Econom. 225:(2):200–30
     [Crossref] [Google Scholar]
  6. 6.
    Cengiz D, Dube A, Lindner A, Zipperer B. 2019.. The effect of minimum wages on low-wage jobs. . Q. J. Econ. 134:(3):1405–54
     [Crossref] [Google Scholar]
  7. 7.
    Cook AC, Leung G, Smith RA. 2020.. Marijuana decriminalization, medical marijuana laws, and fatal traffic crashes in US cities, 2010–2017. . Am. J. Public Health 110:(3):363–69
     [Crossref] [Google Scholar]
  8. 8.
    de Chaisemartin C, D'Haultfoeuille X. 2018.. Fuzzy differences-in-differences. . Rev. Econ. Stud. 85:(2):999–1028
     [Crossref] [Google Scholar]
  9. 9.
    de Chaisemartin C, D'Haultfoeuille X. 2020.. Two-way fixed effects estimators with heterogeneous treatment effects. . Am. Econ. Rev. 110:(9):2964–96
     [Crossref] [Google Scholar]
  10. 10.
    de Chaisemartin C, D'Haultfoeuille X. 2022.. Difference-in-differences estimators of intertemporal treatment effects. NBER Work. Pap. 29873
  11. 11.
    de Chaisemartin C, D'Haultfoeuille X. 2023.. Two-way fixed effects and differences-in-differences with heterogeneous treatment effects: a survey. . Econom. J. 26:(3):C1–30
     [Crossref] [Google Scholar]
  12. 12.
    Deshpande M, Li Y. 2019.. Who is screened out? Application costs and the targeting of disability programs. . Am. Econ. J. Econ. Policy 11:(4):213–48
     [Crossref] [Google Scholar]
  13. 13.
    Dube A, Girardi D, Jordà S, Taylor AM. 2023.. A local projections approach to difference-in-differences event studies. NBER Work. Pap. 31184
  14. 14.
    Gardner J. 2022.. Two-stage differences in differences. . arXiv:2207.05943 [econ]
  15. 15.
    Goodman-Bacon A. 2018.. Difference-in-differences with variation in treatment timing. NBER Work. Pap. 25018
  16. 16.
    Goodman-Bacon A. 2021.. Difference-in-differences with variation in treatment timing. . J. Econom. 225:(2):254–77
     [Crossref] [Google Scholar]
  17. 17.
    Gupta S, Montenovo L, Nguyen T, Lozano-Rojas F, Schmutte I, et al. 2023.. Effects of social distancing policy on labor market outcomes. . Contemp. Econ. Policy 41:(1):166–93
     [Crossref] [Google Scholar]
  18. 18.
    Hollingsworth A, Karbownik K, Thomasson MA, Wray A. 2022.. The gift of a lifetime: the hospital, modern medicine, and mortality. NBER Work. Pap. 30663
  19. 19.
    Jayachandran S, Lleras-Muney A, Smith KV. 2010.. Modern medicine and the twentieth century decline in mortality: evidence on the impact of sulfa drugs. . Am. Econ. J. Appl. Econ. 2:(2):118–46
     [Crossref] [Google Scholar]
  20. 20.
    Kim NH, Elani HW, Kawachi I. 2023.. Did dental insurance expansion improve dental care needs among Korean adults? Difference in difference analysis. . J. Epidemiol. 33:(2):101–8
     [Crossref] [Google Scholar]
  21. 21.
    Mullachery PH, Quistberg DA, Lazo M, Indvik K, Perez-Ferrer C, et al. 2022.. Evaluation of the national sobriety checkpoints program in Mexico: a difference-in-difference approach with variation in timing of program adoption. . Inj. Epidemiol. 9:(1):32
     [Crossref] [Google Scholar]
  22. 22.
    Pei Z, Pischke JS, Schwandt H. 2019.. Poorly measured confounders are more useful on the left than on the right. . J. Bus. Econ. Stat. 37:(2):205–16
     [Crossref] [Google Scholar]
  23. 23.
    Rambachan A, Roth J. 2023.. A more credible approach to parallel trends. . Rev. Econ. Stud. 90:(5):2555–91
     [Crossref] [Google Scholar]
  24. 24.
    Roth J. 2022.. Pretest with caution: event-study estimates after testing for parallel trends. . Am. Econ. Rev. Insights 4:(3):305–22
     [Crossref] [Google Scholar]
  25. 25.
    Roth J, Sant'Anna PHC, Bilinski A, Poe J. 2023.. What's trending in difference-in-differences? A synthesis of the recent econometrics literature. . J. Econom. 235:(2):2218–44
     [Crossref] [Google Scholar]
  26. 26.
    Shahn Z, Dukes O, Richardson D, Tchetgen E, Robins J. 2023.. Structural nested mean models under parallel trends assumptions. . arXiv:2204.10291 [stat.ME]
  27. 27.
    Sun L, Abraham S. 2021.. Estimating dynamic treatment effects in event studies with heterogeneous treatment effects. . J. Econom. 225:(2):175–99
     [Crossref] [Google Scholar]
  28. 28.
    Thomasson MA, Treber J. 2008.. From home to hospital: the evolution of childbirth in the United States, 1928–1940. . Explor. Econ. Hist. 45:(1):76–99
     [Crossref] [Google Scholar]
  29. 29.
    Wing C, Freedman SM, Hollingsworth A. 2024.. Stacked difference-in-differences. NBER Work. Pap. 32054. https://doi.org/10.3386/w32054
  30. 30.
    Wing C, Simon K, Bello-Gomez RA. 2018.. Designing difference in difference studies: best practices for public health policy research. . Annu. Rev. Public Health 39::453–69
     [Crossref] [Google Scholar]
  31. 31.
    Wooldridge JM. 2005.. Fixed-effects and related estimators for correlated random-coefficient and treatment-effect panel data models. . Rev. Econ. Stat. 87:(2):385–90
     [Crossref] [Google Scholar]
  32. 32.
    Wooldridge JM. 2021.. Two-way fixed effects, the two-way Mundlak regression, and difference-in-differences estimators. Work. Pap. , Mich. State Univ., East Lansing
  33. 33.
    Yan Y, Yoshihama M, Hong JS, Jia F. 2023.. Substance use among Asian American adults in 2016–2020: a difference-in-difference analysis of a national survey on drug use and health data. . Am. J. Public Health 113:(6):671–79
     [Crossref] [Google Scholar]
/content/journals/10.1146/annurev-publhealth-061022-050825
Loading
Designing Difference-in-Difference Studies with Staggered Treatment Adoption: Key Concepts and Practical Guidelines
Annual Review of Public Health 45, 485 (2024); https://doi.org/10.1146/annurev-publhealth-061022-050825
/content/journals/10.1146/annurev-publhealth-061022-050825
/content/journals/10.1146/annurev-publhealth-061022-050825
Loading

Data & Media loading...

  • Article Type: Review Article

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error