How Much Should We Trust Differences-in-Differences Estimates?
Causal Question / Estimand
Not a new estimand but a critique of inference for the standard DiD treatment effect (Causal-Estimand): when outcomes are serially correlated over many periods, are the conventional standard errors trustworthy?
Identification Strategy
A simulation-based audit, not a new identification result. The authors generate placebo laws at random in real CPS state-level female-wage data and estimate the DiD “effect” of these non-events. Under correct inference, a 5%-level test should reject 5% of the time; instead it rejects up to 45% of the time — conventional DiD standard errors are badly understated.
Key Assumptions
Takes Parallel-Trends as given (the placebos satisfy it by construction) and isolates the serial-correlation problem: the OLS/DiD point estimate is fine, but i.i.d.-style variance estimates ignore within-unit autocorrelation, violating the independence assumptions behind the usual standard errors.
Threats to Validity
n/a — the paper is about a threat (to inference). Its own caveat: the best fix depends on the number of clusters. Parametric AR corrections perform poorly; the block bootstrap needs many states; collapsing the time series to a single pre/post pair, or clustering at the unit level, works even with few clusters.
Setting / Data
Monte Carlo experiments on Current Population Survey state-by-year female wages (1979–1999), plus simulations varying the number of states and time periods.
Key Claims
- Multi-period DiD with serially correlated outcomes produces severely over-rejecting tests under naive standard errors.
- Parametric time-series corrections do not reliably fix it.
- Cluster-robust standard errors (clustering on the unit) and collapsing the pre/post time dimension are simple, effective remedies — provided enough clusters.
Connections
- See also: Meyer1995-NaturalAndQuasiExperiments (design/inference care), BakerEtAl2025-DiDPractitionerGuide (modern inference guidance), DiD
- A foundational entry in the “how much should we trust DiD” lineage continued by BakerLarckerWang2022-HowMuchTrustStaggeredDiD
Citation
Bertrand, M., Duflo, E., & Mullainathan, S. (2004). How Much Should We Trust Differences-in-Differences Estimates? The Quarterly Journal of Economics, 119(1), 249–275. https://doi.org/10.1162/003355304772839588