In Praise of Moderation
Causal Question / Estimand
Methodological/normative: what role should pre-analysis plans play in randomized evaluations, and how binding should they be?
Identification Strategy
Not an identification paper. Takes randomized experimental design as given and asks how PAPs should be scoped. Argues for moderation: PAPs are valuable for constraining specification search and selective reporting (protecting the credibility of confirmatory tests), but overly rigid or exhaustive plans are costly and can suppress legitimate, data-driven learning and exploratory analysis. Distinguishes confirmatory analyses (which a PAP should bind tightly) from exploratory ones (which it should leave open and label as such), and discusses what registries can and cannot enforce.
Key Assumptions
n/a — theoretical/normative methodological piece. Presumes the researcher-degrees-of-freedom problem the Pre-Analysis-Plan is designed to mitigate.
Threats to Validity
n/a — theoretical contribution. Identifies failure modes of PAP practice: over-specification, false comfort, and crowding out of valuable exploration; versus under-use that leaves p-hacking unchecked.
Setting / Data
n/a — theoretical, with reference to the AEA RCT Registry and economics experimental practice.
Key Claims
- PAPs meaningfully bind only confirmatory hypothesis tests; exploratory work should be pre-labelled as such rather than forced into the plan.
- Excessively detailed PAPs are costly and can reduce, not increase, the scientific value of a study.
- Registration plus a moderately scoped PAP is the practical optimum.
Connections
- Develops the Pre-Analysis-Plan concept; situated within the transparency agenda of ChristensenMiguel2018-TransparencyReproducibility.
- Concerns the conduct of RCTs analysed in AtheyImbens2017-EconometricsOfRandomizedExperiments. See also RCT
Citation
Duflo, E., Banerjee, A., Finkelstein, A., Katz, L. F., Olken, B. A., & Sautmann, A. (2020). In Praise of Moderation: Suggestions for the Scope and Use of Pre-Analysis Plans for RCTs in Economics. NBER Working Paper No. 26993.