Photometric LSST Astronomical Time-series Classification Challenge PLAsTiCC: Selection of a Performance Metric for Classification Probabilities Balancing Diverse Science Goals

Creators: Malz, A. I.; Hložek, R.; Allam, T., Jr.; Bahmanyar, A.; Biswas, R.; Dai, M.; Galbany, L.; Ishida, E. E. O.; Jha, S. W.; Jones, D. O.; Kessler, R.; Lochner, M.; Mahabal, A. A.; Mandel, K. S.; Martínez-Galarza, J. R.; McEwen, J. D.; Muthukrishna, D.; Narayan, G.; Peiris, H.; Peters, C. M.; Ponder, K.; Setzer, C. N.; LSST Dark Energy Science Collaboration; LSST Transients and Variable Stars Science Collaboration

Abstract

Classification of transient and variable light curves is an essential step in using astronomical observations to develop an understanding of the underlying physical processes from which they arise. However, upcoming deep photometric surveys, including the Large Synoptic Survey Telescope (LSST), will produce a deluge of low signal-to-noise data for which traditional type estimation procedures are inappropriate. Probabilistic classification is more appropriate for such data but is incompatible with the traditional metrics used on deterministic classifications. Furthermore, large survey collaborations like LSST intend to use the resulting classification probabilities for diverse science objectives, indicating a need for a metric that balances a variety of goals. We describe the process used to develop an optimal performance metric for an open classification challenge that seeks to identify probabilistic classifiers that can serve many scientific interests. The Photometric LSST Astronomical Time-series Classification Challenge (PLAsTiCC) aims to identify promising techniques for obtaining classification probabilities of transient and variable objects by engaging a broader community beyond astronomy. Using mock classification probability submissions emulating realistically complex archetypes of those anticipated of PLAsTiCC, we compare the sensitivity of two metrics of classification probabilities under various weighting schemes, finding that both yield results that are qualitatively consistent with intuitive notions of classification performance. We thus choose as a metric for PLAsTiCC a weighted modification of the cross-entropy because it can be meaningfully interpreted in terms of information content. Finally, we propose extensions of our methodology to ever more complex challenge goals and suggest some guiding principles for approaching the choice of a metric of probabilistic data products.

Additional Information

© 2019 The American Astronomical Society. Received 2018 October 18; revised 2019 July 31; accepted 2019 August 8; published 2019 October 10. Author contributions are listed below. A.I.M.: conceptualization, data curation, formal analysis, investigation, methodology, project administration, software, supervision, validation, visualization, writing—editing, writing—original draft R.H.: data curation, formal analysis, funding acquisition, investigation, project administration, software, supervision, validation, visualization, writing—editing, writing—original draft T.A.Jr.: investigation, software, validation, writing—original draft A.B.: formal analysis, investigation, methodology, software, writing—editing, writing—original draft R.B.: conceptualization, methodology, software, supervision, writing—editing, writing—original draft M.D.: writing—editing L.G.: writing—editing E.E.O.I.: conceptualization, project administration, supervision, writing—editing S.W.J.: writing—editing D.J.: software R.K.: writing—editing M.L.: conceptualization, data curation, formal analysis, visualization, writing—editing A.A.M.: data curation, software, writing—editing, writing—original draft K.S.M.: conceptualization, supervision, writing—editing J.R.M-G.: data curation, software, visualization, writing—original draft J.D.M.: conceptualization, investigation, supervision D.M.: data curation, validation G.N.: data curation, formal analysis H.P.: conceptualization, funding acquisition, supervision C.M.P.: writing—editing K.P.: visualization, writing—editing C.N.S.: conceptualization, software. This paper has undergone internal review in the LSST Dark Energy Science Collaboration. The authors would like to thank Melissa Graham, Weikang Lin, and Chad Schafer for serving as the LSST-DESC publication review committee, as well as Tom Loredo for other helpful feedback. The authors also express gratitude to the anonymous referee for substantive suggestions that improved the paper. A.I.M. was advised by David W. Hogg and was supported by National Science Foundation grant AST-1517237. A.I.M. acknowledges support from the Max Planck Society and the Alexander von Humboldt Foundation in the framework of the Max Planck-Humboldt Research Award endowed by the Federal Ministry of Education and Research. T.A. is supported in part by STFC. R.B. and C.S. are supported by the Swedish Research Council (VR) through the Oskar Klein Centre. Their work was further supported by the research environment grant "Gravitational Radiation and Electromagnetic Astrophysical Transients (GREAT)" funded by the Swedish Research council (VR) under Dnr 2016-06012. A.A.M. was supported in part by the NSF grants AST-0909182, AST-1313422, AST-1413600, and AST-1518308, and by the Ajax Foundation. E. E. O. I. acknowledges support from CNRS 2017 MOMENTUM grant. D.O.J. is supported by a Gordon and Betty Moore Foundation postdoctoral fellowship at the University of California, Santa Cruz. L.G. was funded by the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 839090. The financial assistance of the National Research Foundation (NRF) toward this research is hereby acknowledged. Opinions expressed and conclusions arrived at, are those of the authors and are not necessarily to be attributed to the NRF. This work is partially supported by the European Research Council under the European Communitys Seventh Framework Programme (FP7/2007-2013)/ERC grant agreement no 306478-CosmicDawn. Canadian co-authors acknowledge support from the Natural Sciences and Engineering Research Council of Canada. The Dunlap Institute is funded through an endowment established by the David Dunlap family and the University of Toronto. The authors at the University of Toronto acknowledge that the land on which the University of Toronto is built is the traditional territory of the Haudenosaunee, and most recently, the territory of the Mississaugas of the New Credit First Nation. They are grateful to have the opportunity to work in the community, on this territory. We acknowledge the University of Chicago Research Computing Center for support of this work. This research used resources of the National Energy Research Scientific Computing Center (NERSC), a U.S. Department of Energy Office of Science User Facility operated under Contract No. DE-AC02-05CH11231. This research at Rutgers University is supported by US Department of Energy award DE-SC0011636. The DESC acknowledges ongoing support from the Institut National de Physique Nucléaire et de Physique des Particules in France; the Science & Technology Facilities Council in the United Kingdom; and the Department of Energy, the National Science Foundation, and the LSST Corporation in the United States. DESC uses resources of the IN2P3 Computing Center (CC-IN2P3–Lyon/Villeurbanne—France) funded by the Centre National de la Recherche Scientifique; the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231; STFC DiRAC HPC Facilities, funded by UK BIS National E-infrastructure capital grants; and the UK particle physics grid, supported by the GridPP Collaboration. This work was performed in part under DOE Contract DE-AC02-76SF00515. Software: jupyter (Kluyver et al. 2016), matplotlib (Hunter 2007), numpy (Oliphant 2006, 2007; van der Walt et al. 2011), proclam (Malz 2018), scikit-learn (Pedregosa et al. 2011), scipy (Jones et al. 2001; Buitinck et al. 2013).

Attached Files

Published - Malz_2019_AJ_158_171.pdf

Submitted - 1809.11145.pdf

Files

Malz_2019_AJ_158_171.pdf

Files (2.9 MB)

Name	Size	Download all
Malz_2019_AJ_158_171.pdf md5:a9bac2fe8b1428c8767b541e0ea9a8ff	1.5 MB	Preview Download
1809.11145.pdf md5:d285b6587b2993a4b552540e754424df	1.5 MB	Preview Download

Additional details

	All versions	This version
Views	0	0
Downloads	0	0
Data volume	0 Bytes	0 Bytes