Megastudies improve the impact of applied behavioural science
- Creators
- Milkman, Katherine L.
- Gromet, Dena
- Ho, Hung
- Kay, Joseph S.
- Lee, Timothy W.
- Pandiloski, Pepi
- Park, Yeji
- Rai, Aneesh
- Bazerman, Max
- Beshears, John
- Bonacorsi, Lauri
- Camerer, Colin
- Chang, Edward
- Chapman, Gretchen
- Cialdini, Robert
- Dai, Hengchen
- Eskreis-Winkler, Lauren
- Fishbach, Ayelet
- Gross, James J.
- Horn, Samantha
- Hubbard, Alexa
- Jones, Steven J.
- Karlan, Dean
- Kautz, Tim
- Kirgios, Erika
- Klusowski, Joowon
- Kristal, Ariella
- Ladhania, Rahul
- Loewenstein, George
- Ludwig, Jens
- Mellers, Barbara
- Mullainathan, Sendhil
- Saccardo, Silvia
- Spiess, Jann
- Suri, Gaurav
- Talloen, Joachim H.
- Taxer, Jamie
- Trope, Yaacov
- Ungar, Lyle
- Volpp, Kevin G.
- Whillans, Ashley
- Zinman, Jonathan
- Duckworth, Angela L.
Abstract
Policy-makers are increasingly turning to behavioural science for insights about how to improve citizens' decisions and outcomes. Typically, different scientists test different intervention ideas in different samples using different outcomes over different time intervals. The lack of comparability of such individual investigations limits their potential to inform policy. Here, to address this limitation and accelerate the pace of discovery, we introduce the megastudy—a massive field experiment in which the effects of many different interventions are compared in the same population on the same objectively measured outcome for the same duration. In a megastudy targeting physical exercise among 61,293 members of an American fitness chain, 30 scientists from 15 different US universities worked in small independent teams to design a total of 54 different four-week digital programmes (or interventions) encouraging exercise. We show that 45% of these interventions significantly increased weekly gym visits by 9% to 27%; the top-performing intervention offered microrewards for returning to the gym after a missed workout. Only 8% of interventions induced behaviour change that was significant and measurable after the four-week intervention. Conditioning on the 45% of interventions that increased exercise during the intervention, we detected carry-over effects that were proportionally similar to those measured in previous research. Forecasts by impartial judges failed to predict which interventions would be most effective, underscoring the value of testing many ideas at once and, therefore, the potential for megastudies to improve the evidentiary value of behavioural science.
Additional Information
© 2021 Nature Publishing Group. Received 22 October 2020; Accepted 13 October 2021; Published 08 December 2021. Support for this research was provided in part by the Robert Wood Johnson Foundation, the AKO Foundation, J. Alexander, M. J. Leder, W. G. Lichtenstein, the Pershing Square Fund for Research on the Foundations of Human Behavior from Harvard University and by Roybal Center grants (P30AG034546 and 5P30AG034532) from the National Institute on Aging. The views expressed here do not necessarily reflect the views of any of these individuals or entities. We thank 24 Hour Fitness for partnering with the Behavior Change for Good Initiative at the University of Pennsylvania to make this research possible. Data availability: The data analysed in this paper were provided by 24 Hour Fitness and we have their legal permission to share the deidentified data. We have therefore made deidentified data available at https://osf.io/9av87/?view_only=8bb9282111c24f81a19c2237e7d7eba3. Furthermore, tables of all of the preregistration links for each of the substudies with the interventions and the prediction studies are available in Supplementary Tables 2 and 30. Code availability: The code to replicate the analyses and figures in the paper and Supplementary Information is available online (https://osf.io/9av87/?view_only=8bb9282111c24f81a19c2237e7d7eba3). Author Contributions: K.L.M., D.G., A.R., M.B., J.B., L.B., E.C., G.C., R.C., H.D., L.E.-W., A.F., J.J.G., S.H., A.H., S.J.J., D.K., E.K., J.K., A.K., G.L., B.M., S.M., S.S., G.S., J.H.T., J.T., Y.T., L.U., K.G.V., A.W., J.Z. and A.L.D. designed the research. K.L.M., D.G., J.S.K., P.P., Y.P., A.L.D. and A.R. performed the research. H.H., T.W.L., P.P. and Y.P. analysed the data. K.L.M. and A.L.D wrote the paper. D.G., H.H., J.S.K., T.W.L., P.P., Y.P., A.R., M.B., J.B., C.C., G.C., H.D., A.F., J.J.G., D.K., T.K., E.K., J.K., R.L., J.L., B.M., S.M., S.S., J.S., A.W. and J.Z. provided feedback on the paper. K.L.M., D.G., J.S.K., T.K., R.L. and S.M. supervised data analysis. K.L.M., D.G., H.H., J.S.K. and T.W.L. prepared the Supplementary Information. The authors declare no competing interests. The authors did not receive commercial benefits from the fitness chain or speaking/consulting fees related to any of the interventions presented here. Peer review information: Nature thanks Charles Shearer and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.Attached Files
Accepted Version - nihms-1774681.pdf
Supplemental Material - 41586_2021_4128_Fig2_ESM.webp
Supplemental Material - 41586_2021_4128_MOESM1_ESM.pdf
Supplemental Material - 41586_2021_4128_MOESM2_ESM.pdf
Supplemental Material - 41586_2021_4128_Tab1_ESM.jpg
Supplemental Material - 41586_2021_4128_Tab2_ESM.jpg
Supplemental Material - 41586_2021_4128_Tab3_ESM.jpg
Supplemental Material - 41586_2021_4128_Tab4_ESM.jpg
Supplemental Material - 41586_2021_4128_Tab5_ESM.jpg
Supplemental Material - 41586_2021_4128_Tab6_ESM.jpg
Supplemental Material - 41586_2021_4128_Tab7_ESM.jpg
Supplemental Material - 41586_2021_4128_Tab8_ESM.jpg
Supplemental Material - 41586_2021_4128_Tab9_ESM.jpg
Files
Name | Size | Download all |
---|---|---|
md5:b6291475e4519a5de1acb904280c83a4
|
135.7 kB | Preview Download |
md5:c6bed114608c27ce7ffa9dc188e610f1
|
136.3 kB | Preview Download |
md5:734613d3bb39e033befb86d5b6ea7b91
|
701 Bytes | Preview Download |
md5:c748a1e93af82807fdfcc5c0f587d6e6
|
708 Bytes | Preview Download |
md5:85782b0433352d74816711331ab36ec8
|
136.6 kB | Preview Download |
md5:7f0e4bca8d2fadfc52f9deac26441780
|
4.0 MB | Preview Download |
md5:88d076ae886e7f1bed31e3ca94a49040
|
134.6 kB | Preview Download |
md5:4523ca0b558609bc4bc25bee6419e5e2
|
135.4 kB | Preview Download |
md5:d3c14c686af80d51c0ef0086e432fab6
|
1.5 MB | Preview Download |
md5:547d95c747fc5683e0e8f3b2ffb33316
|
271.3 kB | Download |
md5:0a3b3bb08623392c0b7e95341fb078e9
|
84.7 kB | Preview Download |
md5:c1e63f3fff0451229348def45c111611
|
32.4 kB | Preview Download |
md5:705ef4f64a6a3b3f545b39a906b9af52
|
136.6 kB | Preview Download |
Additional details
- PMCID
- PMC8822539
- Eprint ID
- 112488
- DOI
- 10.1038/s41586-021-04128-4
- Resolver ID
- CaltechAUTHORS:20211216-65471400
- Robert Wood Johnson Foundation
- AKO Foundation
- J. Alexander
- M. J. Leder
- W. G. Lichtenstein
- Harvard University
- Roybal Center
- P30AG034546
- NIH
- 5P30AG034532
- NIH
- Created
-
2021-12-17Created from EPrint's datestamp field
- Updated
-
2023-07-21Created from EPrint's last_modified field