In Pursuit of Balance: Randomization in Practice
Causal Question / Estimand
Not a single empirical estimand but a methodological question: how should researchers randomize in small-sample field experiments to obtain credible, well-balanced estimates of the ATE?
Identification Strategy
Takes Randomization as the identification foundation and asks how design choices affect finite-sample performance. Compares pure randomization against methods that improve covariate balance — stratification (blocking), pairwise matching, and re-randomization — via simulations and a survey of published experiments. Because balancing on covariates changes the sampling distribution, the analysis must control for the variables used to stratify; otherwise standard errors are wrong.
Key Assumptions
Randomization, SUTVA. The methodological point: valid inference requires the estimation model to reflect the randomization design (stratification dummies included).
Threats to Validity
Chance imbalance in small samples; failing to account for the design in analysis (invalid standard errors); under-powered designs (Statistical-Power); opaque reporting of how randomization was actually done.
Setting / Data
Survey of randomization practice in development field experiments (sample sizes often 100–500), plus Monte Carlo simulations.
Key Claims
- With small samples, stratifying or matching on baseline covariates (especially strong predictors of the outcome) improves balance and precision over pure randomization.
- Researchers must include the stratification variables in the analysis; many published papers fail to describe or account for their randomization.
- Re-randomization and pairwise matching help but complicate inference; design and analysis must be consistent.
Connections
- Builds on: Randomization, AtheyImbens2017-EconometricsOfRandomizedExperiments (stratified/paired designs)
- Design choices interact with Statistical-Power (Lenth2001-SampleSizeDetermination).
- Part of the experimental-credibility agenda with ChristensenMiguel2018-TransparencyReproducibility. See also RCT
Citation
Bruhn, M., & McKenzie, D. (2009). In Pursuit of Balance: Randomization in Practice in Development Field Experiments. American Economic Journal: Applied Economics, 1(4), 200–232.