Published June 4, 2020
| Supplemental Material + Submitted
Journal Article
Open
Variability in the analysis of a single neuroimaging dataset by many teams
- Creators
-
Botvinik-Nezer, Rotem
-
Camerer, Colin F.
Abstract
Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.
Additional Information
© 2020 The Author(s), under exclusive licence to Springer Nature Limited. Received 14 November 2019; Accepted 07 April 2020; Published 20 May 2020. Neuroimaging data collection, performed at Tel Aviv University, was supported by the Austrian Science Fund (P29362-G27), the Israel Science Foundation (ISF 2004/15 to T. Schonberg) and the Swedish Foundation for Humanities and Social Sciences (NHS14-1719:1). Hosting of the data on OpenNeuro was supported by a National Institutes of Health (NIH) grant (R24MH117179). We thank M. C. Frank, Y. Assaf and N. Daw for comments on an earlier draft; the Texas Advanced Computing Center for providing computing resources for preprocessing of the data; the Stanford Research Computing Facility for hosting the data; and D. Roll for assisting with data processing. T. Schonberg thanks The Alfredo Federico Strauss Center for Computational Neuroimaging at Tel Aviv University; A.D. thanks the Knut and Alice Wallenberg Foundation and the Marianne and Marcus Wallenberg Foundation (A.D. is a Wallenberg Scholar), the Austrian Science Fund (FWF, SFB F63) and the Jan Wallander and Tom Hedelius Foundation (Svenska Handelsbankens Forskningsstiftelser); F. Holzmeister, J. Huber and M. Kirchler thank the Austrian Science Fund (FWF, SFB F63); D.W. was supported by the Research Foundation Flanders (FWO) and the European Union's Horizon 2020 research and innovation programme (https://ec.europa.eu/programmes/horizon2020/en) under the Marie Skłodowska-Curie grant agreement no. 665501; L. Tisdall was supported by the University of Basel Research Fund for Junior Researchers; C.B.C. was supported by grant 12O7719N from the Research Foundation Flanders; E.L. was supported by grant 12T2517N from the Research Foundation Flanders and Marie Skłodowska-Curie Actions under COFUND grant agreement 665501; A. Eed was supported by a predoctoral fellowship La Caixa-Severo Ochoa from Obra Social La Caixa and also acknowledges Comunidad de Cálculo Científico del CSIC for the high-performance computing (HPC) use; C.L. was supported by the Vienna Science and Technology Fund (WWTF VRG13-007) and Austrian Science Fund (FWF P 32686); A.B.L.V. was supported by the Vienna Science and Technology Fund (WWTF VRG13-007); L.Z. was supported by the Vienna Science and Technology Fund (WWTF VRG13-007), the National Natural Science Foundation of China (no. 71801110), MOE (Ministry of Education in China) Project of Humanities and Social Sciences (no. 18YJC630268) and China Postdoctoral Science Foundation (no. 2018M633270); D.P. is currently supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy 'Science of Intelligence' (EXC 2002/1, project number 390523135); P.H. was supported in part by funding provided by Brain Canada, in partnership with Health Canada, for the Canadian Open Neuroscience Platform initiative; J.-B.P. was partially funded by the NIH (NIH-NIBIB P41 EB019936 (ReproNim), NIH-NIMH R01 MH083320 (CANDIShare) and NIH RF1 MH120021 (NIDM)) and the National Institute Of Mental Health of the NIH under award number R01MH096906 (Neurosynth), as well as the Canada First Research Excellence Fund, awarded to McGill University for the Healthy Brains for Healthy Lives initiative and the Brain Canada Foundation with support from Health Canada; S.B.E. was supported by the European Union's Horizon 2020 Research and Innovation Programme under grant agreement no. 785907 (HBP SGA2); G.M. was supported by the Max Planck Society; S. Heunis has received funding from the Dutch foundation LSH-TKI (grant LSHM16053-SGF); J.F.G.M. was supported by a Graduate Research Fellowship from the NSF and T32 Predoctoral Fellowship from the NIH; B.M. was supported by the Deutsche Forschungsgemeinschaft (grant CRC1193, subproject B01); A.R.L. was supported by NSF 1631325 and NIH R01 DA041353; M.E.H., T.J. and D.J.W. were supported by the Australian National Imaging Facility, a National Collaborative Research Infrastructure Strategy (NCRIS) capability; P.M.I. was supported by VIDI grant 452-17-013 from the Netherlands Organisation for Scientific Research; B.M.B. was supported by the Max Planck Society; J.P.H. was supported by a grant from the Swedish Research Council; R.W.C. and R.C.R. were supported by NIH IRP project number ZICMH002888; D.M.N., R.W.C., and R.C.R. used the computational resources of the National Institutes of Health High Performance Computing Biowulf cluster (http://hpc.nih.gov); D.M.N. was supported by NIH IRP project number ZICMH002960; C.F.C. was supported by the Tianqiao and Chrissy Center for Social and Decision Neuroscience Center Leadership Chair; R.G.B. was supported by the Max Planck Society; R.M.W.J.B. was supported by the Max Planck Society; M.B., O.C. and R.G. were supported by the Belgian Excellence of Science program (EOS project 30991544) from the FNRS-Belgium; O.C. is a research associate at the FRS-FNRS of Belgium; A.D.L. was supported by grant R4195 "Repimpact" of EraNET Neuron; Q.S. was funded by grant no. 71971199,71602175 and 71942004 from the National Natural Science Foundation of China and no. 16YJC630103 of the Ministry of Education of Humanities and Social Science; and T.E.N. was supported by the Wellcome Trust award 100309/Z/12/Z. Data availability: The full fMRI dataset is publicly available on OpenNeuro (https://doi.org/10.18112/openneuro.ds001734.v1.0.4) and is described in detail in a Data Descriptor1. The results reported by all teams are presented in Extended Data Table 2. A table describing the methods used by the analysis teams is available with the analysis code. NeuroVault collections containing the submitted statistical maps are available via the links provided in Extended Data Table 3a. Source data for Figs. 1, 2 are provided with the paper. Readers may obtain access to the data and run the full analysis stream on the team submissions by following the directions at https://github.com/poldrack/narps/tree/master/ImageAnalyses. Access to the raw data requires specifying a URL for the dataset, which is: https://zenodo.org/record/3528329/files/narps_origdata_1.0.tgz. Results (automatically generated figures, results and output logs) for image analyses are available for anonymous download at https://doi.org/10.5281/zenodo.3709275. Code availability: Code for all analyses of the reports and statistical maps submitted by the analysis teams is openly shared in GitHub (https://github.com/poldrack/narps). Image-analysis code was implemented within a Docker container, with software versions pinned for reproducible execution (https://hub.docker.com/r/poldrack/narps-analysis/tags). Python code was automatically tested for quality using the flake8 static analysis tool and the codacy.com code quality assessment tool, and the results of the image-analysis workflow were validated using simulated data. The image-analysis code was independently reviewed by an expert who was not involved in writing the original code. Prediction market analyses were performed using R v.3.6.1; packages were installed using the checkpoint package, which reproducibly installs all package versions as of a specified date (13 August 2019). Analyses reported in this manuscript were performed using code release v.2.0.3 (https://doi.org/10.5281/zenodo.3709273). Although not required to, several analysis teams publicly shared their analysis code. Extended Data Table 3d includes these teams along with the link to their code. Author Contributions: NARPS management team: R.B.-N., F. Holzmeister, C.F.C., A.D., J. Huber, M.J., M. Kirchler, R.A.P. and T. Schonberg. fMRI dataset (experiment design): R.I., J. Durnez, R.A.P. and T. Schonberg. fMRI dataset (data collection): R.I. and T. Schonberg. fMRI dataset (preprocessing, quality assurance and data sharing): R.B.-N., K.J.G., R.A.P. and T. Schonberg. Analysis teams (recruitment, point of contact and management): R.B.-N., R.A.P. and T. Schonberg. Analysis teams (analysis of the submitted results and statistical maps): R.A.P., T.E.N., J.A.M., J.-B.P., A.P., R.B.-N. and T. Schonberg. Code review: T.G. and K.D. Prediction markets (design and management): F. Holzmeister, C.F.C., A.D., J. Huber, M.J. and M. Kirchler. Prediction markets (analysis): F. Holzmeister, R.B.-N., C.F.C., A.D., J. Huber, M.J., M. Kirchler, S.K., R.A.P. and T. Schonberg. Writing the manuscript: R.B.-N., F. Holzmeister, A.D., J. Huber, M.J., M. Kirchler, T.E.N., R.A.P. and T. Schonberg. Participated as members of analysis teams and reviewed and edited the manuscript: R.A.A., P.A., B.M.B., A. Bajracharya, L.B., S. Ball, M.B., N.B., D.B., J.B., R.G.B., R.M.W.J.B., J.P.B., B.B.B., S.B.-S., T.B., K.L.B., A. Bowring, S. Braem, H.R.B., E.G.B., C.B.C., J.A.C., J.J.C., L.C., E.C.C., Z.J.C., O.C., R.W.C., W.A.C., S.C., K.D., C.P.D., A.D.L., M.R.D., L.D., J.B.D., X.D., E.W.D., E.D., C.L.D., J. Dukart, N.W.D., A. Eed, S.B.E., A. Erhart, L.F., G.M.F., S.F., A.G., R.G., S.G., E.G., J.J.G., S.A.E.G., C.G.-G., K.J.G., C.L.G., M.A.G., J.F.G.M., O.G., S. Hakimi, J.P.H., R.H., G. Handjaras, B.B.H., C.H., P.H., G. Herman, S. Heunis, F. Hoffstaedter, J. Hogeveen, S. Holmes, C.-P.H., S.A.H., M.E.H., V.I., A.D.I., P.M.I., A.I.I., A.J., M.R.J., T.J., M.J.E.J., A.C.J., J.W.K., M. Kassinopoulos, C.K., X.-Z.K., T.R.K., N.E.K., B.A.K., A.R.L., C.L., R.L., N.L., H.L., S.L., A. Leemans, A. Leo, E.L., F.L., M.Y.C.L., P.C.L., E.N.L., S.W.L., A.B.L.V., B.C.L., M.L.M., N.M., T.M., C.M., K.M., J.T.M., H.M., A.S.M.L., B.M., K.N.M., G.M., G.D.M., J.M., T.E.N., D.M.N., G.N., M.P.N., E.O., A.I.O., P.P., K.R.P., J.E.P., D.P., Y.P., S.R., P.A.R.-L., R.C.R., E.R., J.R.R., A.M.R.-T., A.R., T. Salo, G.R.S.-L., E.S.-M., M.L.S., D.H.S., Q.S., M.A.S., J.A.S., K.S., A.S., D.V.S., P.S.-H., S.R.S., S.M.T., B.T., J.N.T., G.T., L. Tisdall, S.H.T., C.T.-S., J.J.T.T., L. Tozzi, V.T., L. Turella, A.E.v.V., T.V., J.M.V., S.V., K.V., M.B.W., W.D.W., S.W., D.J.W., D.W., A.X.-P., E.A.Y., S.Y., R.Y., K.S.L.Y., L.Z., X.Z. and J.E.Z. The authors declare no competing interests.Attached Files
Submitted - 843193.full.pdf
Supplemental Material - 41586_2020_2314_Fig3_ESM.webp
Supplemental Material - 41586_2020_2314_Fig4_ESM.webp
Supplemental Material - 41586_2020_2314_Fig5_ESM.webp
Supplemental Material - 41586_2020_2314_Fig6_ESM.webp
Supplemental Material - 41586_2020_2314_Fig7_ESM.webp
Supplemental Material - 41586_2020_2314_MOESM1_ESM.pdf
Supplemental Material - 41586_2020_2314_MOESM2_ESM.pdf
Supplemental Material - 41586_2020_2314_MOESM3_ESM.csv
Supplemental Material - 41586_2020_2314_MOESM4_ESM.csv
Supplemental Material - 41586_2020_2314_Tab1_ESM.jpg
Supplemental Material - 41586_2020_2314_Tab2_ESM.jpg
Supplemental Material - 41586_2020_2314_Tab3_ESM.jpg
Supplemental Material - 41586_2020_2314_Tab4_ESM.jpg
Supplemental Material - 41586_2020_2314_Tab5_ESM.jpg
Files
41586_2020_2314_Tab5_ESM.jpg
Files
(4.6 MB)
| Name | Size | Download all |
|---|---|---|
|
md5:fad7aaa687cffde3302b7e47b9cc5b0d
|
95.9 kB | Preview Download |
|
md5:e9378ddb08f835283ae8ce8381f4cb54
|
683.4 kB | Preview Download |
|
md5:adab4573fe3b18db8b00c6183944862f
|
59.7 kB | Preview Download |
|
md5:7096ff50788ca48aa8396a6a722cb06f
|
121.9 kB | Preview Download |
|
md5:3f228f30229c7fbd44a99f1e550e9249
|
256.4 kB | Download |
|
md5:9328c6bbaceaef1057c59a32904facc6
|
219.5 kB | Preview Download |
|
md5:ec9f96cb786032176dfac46647c99f35
|
558.8 kB | Download |
|
md5:a82fd3d6ebe0cf5d35b2de8953e8eab3
|
1.3 MB | Preview Download |
|
md5:e3c133307c0f3e72f0c5139ec7df10e7
|
646.9 kB | Download |
|
md5:f47f6ea20be2066c3681ebe2e5808db7
|
1.3 kB | Preview Download |
|
md5:ab235a5ab316eedaa3167704937b6ff0
|
79.7 kB | Preview Download |
|
md5:9957b10814d7a073f34aa2ea89bb804a
|
117.2 kB | Download |
|
md5:5066521f4bf9bae106b479c01340e2aa
|
275.2 kB | Download |
|
md5:23ce890fea5d3efdf75d78f90ce0bc28
|
122.8 kB | Preview Download |
|
md5:d175c0ea643a23dd26d67286de892704
|
46.3 kB | Preview Download |
Additional details
- Eprint ID
- 99892
- Resolver ID
- CaltechAUTHORS:20191118-092814513
- P29362-G27
- FWF Der Wissenschaftsfonds
- 2004/15
- Israel Science Foundation
- NHS14-1719:1
- Swedish Foundation for Humanities and Social Sciences
- R24MH117179
- NIH
- Knut and Alice Wallenberg Foundation
- Marianne and Marcus Wallenberg Foundation
- SFB F63
- FWF Der Wissenschaftsfonds
- Jan Wallander and Tom Hedelius Foundation
- 665501
- Marie Curie Fellowship
- University of Basel
- 12O7719N
- Fonds Wetenschappelijk Onderzoek (FWO)
- 12T2517N
- Fonds Wetenschappelijk Onderzoek (FWO)
- Obra Social La Caixa
- WWTF VRG13-007
- Viennese Science and Technology Fund
- P 32686
- FWF Der Wissenschaftsfonds
- 71801110
- National Natural Science Foundation of China
- 18YJC630268
- Ministry of Education (Taipei)
- 2018M633270
- China Postdoctoral Science Foundation
- EXC 2002/1
- Deutsche Forschungsgemeinschaft (DFG)
- 390523135
- Deutsche Forschungsgemeinschaft (DFG)
- Health Canada
- P41 EB019936
- NIH
- R01 MH083320
- NIH
- RF1 MH120021
- NIH
- R01MH096906
- NIH
- Canada First Research Excellence Fund
- Brain Canada Foundation
- 785907
- European Research Council (ERC)
- Max Planck Society
- LSHM16053-SGF
- Dutch Foundation
- NSF Graduate Research Fellowship
- NIH Predoctoral Fellowship
- CRC1193
- Deutsche Forschungsgemeinschaft (DFG)
- IIS-1631325
- NSF
- R01 DA041353
- NIH
- Australian National Imaging Facility
- 452-17-013
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (NWO)
- Swedish Research Council
- ZICMH002888
- NIH
- ZICMH002960
- NIH
- Tianqiao and Chrissy Chen Institute for Neuroscience
- 30991544
- Belgian Excellence of Science
- Fonds de la Recherche Scientifique (FNRS)
- R4195
- EraNET Neuron
- 71971199
- National Natural Science Foundation of China
- 71602175
- National Natural Science Foundation of China
- 71942004
- National Natural Science Foundation of China
- 16YJC630103
- Ministry of Education (China)
- 100309/Z/12/Z
- Wellcome Trust
- Created
-
2019-11-18Created from EPrint's datestamp field
- Updated
-
2023-06-01Created from EPrint's last_modified field
- Caltech groups
- Tianqiao and Chrissy Chen Institute for Neuroscience