Welcome to the new version of CaltechAUTHORS. Login is currently restricted to library staff. If you notice any issues, please email coda@library.caltech.edu
Published December 5, 2019 | Supplemental Material + Submitted + Published
Journal Article Open

Predicting the replicability of social science lab experiments

Abstract

We measure how accurately replication of experimental results can be predicted by black-box statistical models. With data from four large-scale replication projects in experimental psychology and economics, and techniques from machine learning, we train predictive models and study which variables drive predictable replication. The models predicts binary replication with a cross-validated accuracy rate of 70% (AUC of 0.77) and estimates of relative effect sizes with a Spearman ρ of 0.38. The accuracy level is similar to market-aggregated beliefs of peer scientists [1, 2]. The predictive power is validated in a pre-registered out of sample test of the outcome of [3], where 71% (AUC of 0.73) of replications are predicted correctly and effect size correlations amount to ρ = 0.25. Basic features such as the sample and effect sizes in original papers, and whether reported effects are single-variable main effects or two-variable interactions, are predictive of successful replication. The models presented in this paper are simple tools to produce cheap, prognostic replicability metrics. These models could be useful in institutionalizing the process of evaluation of new findings and guiding resources to those direct replications that are likely to be most informative.

Additional Information

© 2019 Altmejd et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Received: June 20, 2019; Accepted: November 13, 2019; Published: December 5, 2019. Data Availability: All data and code files are available from the OSF project repository (https://osf.io/4fn73/, DOI: 10.17605/OSF.IO/4FN73). Generous support was provided by the Alfred P. Sloan Foundation (G-201513929, to CFC), the Chen Center for Social and Decision Neuroscience (to CFC), the Austrian Science Fund FWF (SFB F63 to MK and JH, P29362-G27 to JH), The Jan Wallander and Tom Hedelius Foundation (P2015-0001:1 to MJ, P14-0214 and P13-0156 to AD), The Knut and Alice Wallenberg Foundation (Wallenberg Academy Fellows grant to AD), The Swedish Foundation for Humanities and Social Sciences (NHS14--1719 to MJ and AD), and Forte (2016-07099 to AA). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The authors have declared that no competing interests exist. We thank Isak Roth and Dylan Manfredi for outstanding research assistance. Author Contributions: Conceptualization: Adam Altmejd. Data curation: Adam Altmejd, Anna Dreber, Eskil Forsell, Juergen Huber, Taisuke Imai, Magnus Johannesson, Michael Kirchler, Gideon Nave. Formal analysis: Adam Altmejd, Eskil Forsell, Taisuke Imai, Gideon Nave. Funding acquisition: Anna Dreber, Magnus Johannesson, Colin Camerer. Investigation: Adam Altmejd. Methodology: Adam Altmejd, Taisuke Imai, Gideon Nave. Project administration: Adam Altmejd. Software: Adam Altmejd. Supervision: Anna Dreber, Magnus Johannesson, Colin Camerer. Validation: Adam Altmejd. Visualization: Adam Altmejd. Writing – original draft: Adam Altmejd. Writing – review & editing: Adam Altmejd, Taisuke Imai, Magnus Johannesson, Michael Kirchler, Gideon Nave, Colin Camerer.

Attached Files

Published - journal.pone.0225826.pdf

Submitted - Predicting_Replication.pdf

Supplemental Material - journal.pone.0225826.s001.pdf

Supplemental Material - journal.pone.0225826.s002.pdf

Supplemental Material - journal.pone.0225826.s003.pdf

Supplemental Material - journal.pone.0225826.s004.pdf

Supplemental Material - journal.pone.0225826.s005.pdf

Supplemental Material - journal.pone.0225826.s006.pdf

Supplemental Material - journal.pone.0225826.s007.eps

Supplemental Material - journal.pone.0225826.s008.eps

Supplemental Material - journal.pone.0225826.s009.eps

Supplemental Material - journal.pone.0225826.s010.eps

Files

Predicting_Replication.pdf
Files (1.8 MB)
Name Size Download all
md5:d0a20e59ae5137414a70d94768d854ca
505.4 kB Preview Download
md5:217fdffd49d5e8852777f65f2544f8de
33.0 kB Preview Download
md5:a7e9273bab788e815899408bb8a40649
32.2 kB Preview Download
md5:87b3394e1f1012f8b82e3c783dd96360
38.3 kB Download
md5:3d5f7a934e9e37d9cc5d06cc08ce0dfb
818.7 kB Preview Download
md5:624594d691291eda1a7f4838c76957e1
66.1 kB Preview Download
md5:723e91a724a656f4ceda1ba6353ef8d6
21.7 kB Preview Download
md5:90a1d933c4006560e0e5abb5d58ce166
47.4 kB Preview Download
md5:b63023542a4a5f6ba746a022d2bc46cc
71.0 kB Download
md5:67d9002fe565806f2e41ca0602ada2ae
53.0 kB Download
md5:030bbcd4d7e14c1c061c27c9ed8b13e7
11.5 kB Preview Download
md5:44894c77ccb666ef9a8733cab47a24f4
119.7 kB Download

Additional details

Created:
August 19, 2023
Modified:
October 23, 2023