SBEE seminar series: Stefano DellaVigna
11:45 am -
RCTs to Scale: Comprehensive Evidence from Two Nudge Units
Nudge interventions – behaviorally-motivated design changes with no financial incentives – have quickly expanded from academic studies to larger implementation in so-called Nudge Units in governments. This provides an opportunity to compare interventions in research studies, versus at scale. We assemble a unique data set of 126 RCTs covering over 23 million individuals, including all trials run by two of the largest Nudge Units in the United States.
We compare these trials to a separate sample of nudge trials published in academic journals from two recent meta-analyses. In papers published in academic journals, the average impact of a nudge is very large – an 8.7 percentage point take-up effect, a 33.5% increase over the average control. In the Nudge Unit trials, the average impact is still sizable and highly statistically significant, but smaller at 1.4 percentage points, an 8.1% increase.
We consider five potential channels for this gap: statistical power, selective publication, academic involvement, differences in trial features and in nudge features. Publication bias in the academic journals, exacerbated by low statistical power, can account for the full difference in effect sizes. Academic involvement does not account for the difference. Different features of the nudges, such as in-person versus letter-based communication, likely reflecting institutional constraints, can partially explain the different effect sizes.
We conjecture that larger sample sizes and institutional constraints, which play an important role in our setting, are relevant in other at-scale implementations. Finally, we compare these results to the predictions of academics and practitioners. Most forecasters overestimate the impact for the Nudge Unit interventions, though nudge practitioners are almost perfectly calibrated.
Stefano DellaVigna (2002 PhD, Harvard) is the Daniel Koshland, Sr. Distinguished Professor of Economics and Professor of Business Administration at the University of California, Berkeley. He specializes in Behavioral Economics and has published in international journals such as the American Economic Review, the Journal of Political Economy, and the Quarterly Journal of Economics. He has been a Principal Investigator for NSF Grants (2004-07 and 2016-18), an Alfred P. Sloan Fellow for 2008-10, and is a Distinguished Teaching Award winner (2008). He has been a co-editor of the American Economic Review since 2017. His recent work has focused on (i) the design of model-based field experiments, (ii) the ability of experts to forecast research results, and (iii) the analysis of gender differences in editorial choices and honors in economics.
The Social, Behavioral and Experimental Economics seminar series is a joint presentation of the School of Information, the Ross School of Business and the LSA Department of Economics.
For information on how to watch this lecture and sign up for the SBEE mailing list to receive notice of upcoming events, please visit the SBEE website: https://umbee.github.io/SBEE_Seminars