The Promise and Pitfalls of DiD: Reflections on 16 and Pregnant
Causal Question / Estimand
The DiD treatment effect (Causal-Estimand) when DiD is deployed as a substitute for a failed experiment — anchored in the Kearney–Levine vs Jaeger–Joyce–Kaestner exchange over whether MTV’s 16 and Pregnant reduced teen births (KL: −4.3%; JJK: identification flawed because pre-trends were not parallel).
Identification Strategy
Not a new estimator — a methodological reflection on what DiD identification requires beyond the mechanics. In Potential-Outcomes terms: DiD is only needed because levels differ between groups (equations (1)–(2) of a true experiment fail), so the researcher owes an account of why levels differed and why that same mechanism leaves counterfactual trends untouched (Parallel-Trends, eq. (3)). Since DiD has a regression representation, it cannot be inherently more credible than regression — the same specification and orthogonality burdens apply.
Key Assumptions
Parallel-Trends — with two sharpened caveats: (i) it is functional-form dependent (linear, probit, and logit interaction terms assume different counterfactuals, so the scale must be argued, not defaulted; contra Ai–Norton, interaction coefficients in probit/logit can be the right object); (ii) parallel pre-trends are neither necessary nor sufficient for counterfactual parallel trends. No-Anticipation implicitly in the pre/post framing.
Threats to Validity
The paper is a threat catalogue: unexplained level differences signal mechanisms that plausibly also hit trends; accepting the null of no differential pre-trend is a Type-II-error trap; pre-testing then estimating yields wrong standard errors; functional-form choices smuggle in the counterfactual; and with few groups, group-time error components make naive inference invalid (Cameron–Miller).
Setting / Data
The 16 and Pregnant debate: Kearney–Levine’s DiD/IV design using pre-existing MTV viewership across US media markets, and Jaeger–Joyce–Kaestner’s replication critique — used as a constructive case study rather than an indictment.
Key Claims
- Every DiD paper should explain why treatment and control groups differed in levels, and why that mechanism would not affect trends.
- Parallel trends requires justifying the functional form/scale; the “correct” model (LPM vs probit vs logit) changes the counterfactual.
- Parallel pre-trends ≠ parallel counterfactual trends: not necessary, not sufficient, and pre-testing distorts standard errors.
- “Natural experiment” rhetoric obscures that DiD is a response to the absence of design, not a design-based estimator itself.
Connections
- Anticipates and complements: RothPretrends2022-PretestWithCaution (pre-test distortion and Type-II trap, formalized), RambachanRoth2023-MoreCredibleParallelTrends (bounded violations instead of binary pre-tests)
- Functional-form dependence of parallel trends: AtheyImbens2006-NonlinearDiD (changes-in-changes as the scale-invariant response)
- Level-differences-imply-trend-worries lineage: AshenfelterCard1985-LongitudinalEarnings (Ashenfelter-Dip), Meyer1995-NaturalAndQuasiExperiments
- Pushes back on the framing of: AngristPischke2010-CredibilityRevolution (DiD as “design-based” showpiece)
- Inference with grouped errors: Bertrand2004-HowMuchShouldWeTrustDiD
- See also: DiD
Citation
Kahn-Lang, A., & Lang, K. (2020). The Promise and Pitfalls of Differences-in-Differences: Reflections on 16 and Pregnant and Other Applications. Journal of Business & Economic Statistics, 38(3), 613–620. https://doi.org/10.1080/07350015.2018.1546591