Welcome to the new version of CaltechAUTHORS. Login is currently restricted to library staff. If you notice any issues, please email coda@library.caltech.edu
Published February 23, 2014 | Supplemental Material + Accepted Version
Journal Article Open

Sleep-spindle detection: crowdsourcing and evaluating performance of experts, non-experts and automated methods

Abstract

Sleep spindles are discrete, intermittent patterns of brain activity observed in human electroencephalographic data. Increasingly, these oscillations are of biological and clinical interest because of their role in development, learning and neurological disorders. We used an Internet interface to crowdsource spindle identification by human experts and non-experts, and we compared their performance with that of automated detection algorithms in data from middle- to older-aged subjects from the general population. We also refined methods for forming group consensus and evaluating the performance of event detectors in physiological data such as electroencephalographic recordings from polysomnography. Compared to the expert group consensus gold standard, the highest performance was by individual experts and the non-expert group consensus, followed by automated spindle detectors. This analysis showed that crowdsourcing the scoring of sleep data is an efficient method to collect large data sets, even for difficult tasks such as spindle identification. Further refinements to spindle detection algorithms are needed for middle- to older-aged subjects.

Additional Information

© 2014 Nature America, Inc. Received 17 July 2013. Accepted 31 January 2014. Published online 23 February 2014. The authors thank the Registered Polysomnographic Technologist (RPSGT) experts who participated in the spindle identification task as well as the participants and organizers of the Wisconsin Sleep Cohort who provided the polysomnography data. We also thank C. Liang, E. Leary, H. Ollila and H. Kraemer for their helpful discussions and E.Þ. Ágústsson and H. Moore for their input in the pilot study for this project. We are grateful to the authors of the previously published algorithms who generously shared their code and knowledge about spindle detectors. S.C.W. is supported by the Brain and Behavior Research Foundation and as a Canadian Institutes of Health Research Banting Fellow. E.M. is supported by US National Institutes of Health (NIH) grant NS23724. P.P. received funding from the Caltech SURF program and the NAVY (ONR-MURI N00014-06-1-0734 and UCLA-MURI N00014-10-1-0933). EEG data collection was supported by grants from the National Heart, Lung, and Blood Institute (grant R01HL62252) and the National Center for Research Resources (grant 1UL1RR025011) at the NIH. The authors declare no competing financial interests. Author Contributions: S.C.W., E.M. and P.P. designed the research. P.W. and P.P. designed and coded the Internet interface. S.C.W. and P.W. collected the spindle scoring data. S.C.W. and S.L.W. performed the data analysis. S.L.W. wrote the code to implement the automated spindle detectors. All authors provided input on data analysis and interpretation. P.E.P. also provided source EEG data. H.B.D.S., P.J., E.M. and P.P. also provided financial support. S.C.W., S.L.W. and E.M. wrote the manuscript, which was discussed and edited by all authors.

Attached Files

Accepted Version - nihms562643.pdf

Supplemental Material - nmeth.2855-S1.pdf

Supplemental Material - nmeth.2855-S2.zip

Files

nmeth.2855-S2.zip
Files (5.0 MB)
Name Size Download all
md5:f5598e974b92233fdad3bbf117f0ed7e
17.8 kB Preview Download
md5:3f9a9e104b8efc54cde929439e68cf80
2.1 MB Preview Download
md5:c3a64dcdf42984d824c7176d063cd2f9
2.9 MB Preview Download

Additional details

Created:
August 22, 2023
Modified:
October 26, 2023