Reconciling Conflicting Evidence on the Performance of Propensity-Score Matching Methods

Causal Question / Estimand

Can non-experimental Propensity-Score matching recover the experimental ATT? The paper evaluates matching estimators by benchmarking them against the experimental NSW results (the LaLonde critique), reconciling the optimistic Dehejia–Wahba findings with more pessimistic ones.

Identification Strategy

Uses experimental data combined with non-experimental comparison groups to test the selection-on-observables assumption directly: a credible matching estimator should reproduce the experimental benchmark. They show the apparent success of PSM in Dehejia–Wahba is fragile — sensitive to the chosen subsample, the set of conditioning variables, and the matching algorithm — and that difference-in- differences matching (matching on changes, which differences out time-invariant unobservables) is more robust than cross-sectional matching.

Key Assumptions

Ignorability (the assumption under scrutiny), Overlap (common support), Propensity-Score specification, SUTVA. DiD-matching additionally invokes a conditional Parallel-Trends-type assumption on unobservables.

Threats to Validity

The paper is a threat analysis: estimates swing with sample definition, covariate choice, and algorithm; cross-sectional matching fails to remove time-invariant unobserved confounding; results do not generalize across data subsets.

Setting / Data

LaLonde (1986) NSW experimental data and the Dehejia–Wahba non-experimental comparison samples (CPS/PSID), job-training programme earnings.

Key Claims

• Matching does not automatically overcome LaLonde’s critique; its measured performance is highly specification- and sample-dependent.
• Difference-in-differences (longitudinal) matching outperforms cross-sectional matching by absorbing time-invariant unobservables.
• Common-support imposition and balancing diagnostics materially affect conclusions — reported success can be an artifact of researcher choices.

Connections

• Critiques the optimistic reading of Imbens2004-NonparametricATEReview / Imbens2015-MatchingMethodsInPractice by stress-testing matching empirically.
• Bridges PSM and DiD via DiD-matching (Abadie2005-SemiparametricDiD, Parallel-Trends).
• Motivates the sensitivity analysis emphasized by CaliendoKopeinig2008-PSMImplementationGuidance. See also PSM

Citation

Smith, J. A., & Todd, P. E. (2005). Reconciling Conflicting Evidence on the Performance of Propensity-Score Matching Methods. American Economic Review, 95(2), 112–118. (Summarizing Smith & Todd, Journal of Econometrics, 2005.)