Evaluating replicability of laboratory experiments in economics
Abstract
The reproducibility of scientific findings has been called into question. To contribute data about reproducibility in economics, we replicate 18 studies published in the American Economic Review and the Quarterly Journal of Economics in 2011-2014. All replications follow predefined analysis plans publicly posted prior to the replications, and have a statistical power of at least 90% to detect the original effect size at the 5% significance level. We find a significant effect in the same direction as the original study for 11 replications (61%); on average the replicated effect size is 66% of the original. The reproducibility rate varies between 67% and 78% for four additional reproducibility indicators, including a prediction market measure of peer beliefs.
Additional Information
© 2016 American Association for the Advancement of Science. Received 16 December 2015; accepted 19 February 2016; Published online 3 March 2016. For financial support we thank: Austrian Science Fund FWF (START-grant Y617-G11), Austrian National Bank (grant OeNB 14953), Behavioral and Neuroeconomics Discovery Fund (CFC), Jan Wallander and Tom Hedelius Foundation (P2015-0001:1 and P2013-0156:1), Knut and Alice Wallenberg Foundation (Wallenberg Academy Fellows grant to A. Dreber), Swedish Foundation for Humanities and Social Sciences (NHS14-1719:1), and Sloan Foundation (G-2015-13929). We thank the following experimental labs for kindly allowing us to use them for replication experiments: Center for Behavioral Economics at National University of Singapore, Center for Neuroeconomics Studies at Claremont Graduate University, Frankfurt Laboratory for Experimental Economic Research, Harvard Decision Science Laboratory, Innsbruck ECONLAB, and Nuffield Centre for Experimental Social Sciences. We thank the following persons for assistance with the experiments: Jorge Barraza, Agneta Berge, Rahul Bhui, Andreas Born, Nina Cohodes, Ho Kinh Dat, Christoph Dohmen, Zayan Faiayd, Malte Heissel, Austin Henderson, Gabe Mansur, Jutta Preussler, Lukas Schultze, Garrett Thoelen, and Elizabeth Warner. The data reported in this paper are tabulated in tables S1, S3 and S4 and the Replication Reports, analyses code, and the data from the replications are available at www.experimentaleconreplications.com and at OSF (osf.io/bzm54). The authors report no potential conflicts of interest. No MTAs, patents or patent applications apply to methods or data in the paper. CC, AD, JH, TH, MJ, and MK designed research; CC, AD, EF, JH, TH, MJ, and MK wrote the paper; EF, JA, TC, TH, TP helped design the prediction market part; EF, FH, JH, MK, MR, TP, and HW analyzed data; AA, EH, FH, TI, SI, GN, MR, and HW carried out the replications (including re-estimating the original estimate with the replication data); all authors approved the final manuscript.Attached Files
Supplemental Material - aaf0918-Camerer-SM.pdf
Files
| Name | Size | Download all |
|---|---|---|
|
md5:fbf13d096f05601ccaf0a76373d7a82d
|
1.5 MB | Preview Download |
Additional details
- Eprint ID
- 64988
- DOI
- 10.1126/science.aaf0918
- Resolver ID
- CaltechAUTHORS:20160303-090305606
- Y617-G11
- FWF Der Wissenschaftsfonds
- OeNB 14953
- Austrian National Bank
- Behavioral and Neuroeconomics Discovery Fund (CFC)
- P2015-0001:1
- Jan Wallander and Tom Hedelius Foundation
- P2013-0156:1
- Jan Wallander and Tom Hedelius Foundation
- Knut and Alice Wallenberg Foundation (Wallenberg Academy Fellows)
- NHS14-1719:1
- Swedish Foundation for Humanities and Social Sciences
- G-2015-13929
- Sloan Foundation
- Created
-
2016-03-03Created from EPrint's datestamp field
- Updated
-
2021-11-10Created from EPrint's last_modified field