DiD methods: No single best, performance context-dependent
Federal Reserve economists John Coglianese and Jade A. Fang evaluated the empirical performance of difference-in-differences (DiD) estimators using over 134,000 state-level placebo event studies. Their research finds no single method consistently outperforms others, emphasizing context-dependency.
Context shapes estimator performance
John Coglianese and Jade A. Fang evaluated 13 difference-in-differences (DiD) estimators using over 134,000 state-level placebo event studies.
Their analysis, covering outcomes like unemployment, payroll employment, and house price indices, found no single method consistently outperforms others.
Performance is highly context-dependent, varying across outcomes and states.
Synthetic-control-like methods sometimes outperform and sometimes underperform two-way fixed effects (TWFE)-like and matching methods.
This variability suggests a bias-variance trade-off, where flexible models can construct better counterfactuals but also risk overfitting.
The study also highlights that choices like seasonal adjustment and normalization significantly impact precision, often as much as the estimator itself.
Moreover, the characteristics of the "treated" unit, particularly whether it is a large, diverse state, influence the variance of placebo estimates considerably.
Placebo tests reveal real-world efficiency
The paper advocates for placebo tests as a crucial tool for evaluating difference-in-differences (DiD) methods, especially given the proliferation of new estimators.
Placebo tests, a form of randomization inference, involve repeatedly conducting event studies over thousands of randomly chosen "fake" events.
This approach measures estimator efficiency by observing the spread of placebo estimates, with a narrower distribution indicating better efficiency.
A key advantage over Monte Carlo simulations is their use of real-world data, which circumvents the need for strict assumptions about the data generating process.
This provides a more realistic assessment of estimator performance in empirical settings, complementing econometric theory and simulation results.
Empirical rigor over theoretical dogma
This paper offers a crucial reality check for applied microeconomists, demonstrating that theoretical superiority does not guarantee empirical performance.
Its findings underscore the indispensable role of context-specific validation, urging researchers to move beyond default choices.
For practitioners, rigorous placebo testing is not optional, but fundamental for robust causal inference.