Welcome to the new version of CaltechAUTHORS. Login is currently restricted to library staff. If you notice any issues, please email coda@library.caltech.edu
Published July 2023 | v2
Journal Article Open

The Cell Tracking Challenge: 10 years of objective benchmarking

Abstract

The Cell Tracking Challenge is an ongoing benchmarking initiative that has become a reference in cell segmentation and tracking algorithm development. Here, we present a significant number of improvements introduced in the challenge since our 2017 report. These include the creation of a new segmentation-only benchmark, the enrichment of the dataset repository with new datasets that increase its diversity and complexity, and the creation of a silver standard reference corpus based on the most competitive results, which will be of particular interest for data-hungry deep learning-based strategies. Furthermore, we present the up-to-date cell segmentation and tracking leaderboards, an in-depth analysis of the relationship between the performance of the state-of-the-art methods and the properties of the datasets and annotations, and two novel, insightful studies about the generalizability and the reusability of top-performing methods. These studies provide critical practical conclusions for both developers and users of traditional and machine learning-based cell segmentation and tracking algorithms.

Additional Information

© 2023 The Author(s). This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/. The authors thank J. Padilla Pérez, who worked for many hours on the annotation of the new datasets; J.-Y. Tinevez, who kindly added the CTC measures into the popular TrackMate software and who, with T. Pietzsch, developed the Mastodon software that became instrumental for us when preparing the tracking annotations of the large embryonic datasets; and the participants of the challenge not included in the list of authors of this analysis paper, listed on the challenge website (http://celltrackingchallenge.net/participants/). This work was funded by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación (MCIU/AEI/10.13039/50110011033) and FEDER funds UE under Grants RTI2018-094494-B-C22, TED2021-131300B-I00, PDI2021-122409OB-C22 (C.O.S.); Czech Ministry of Education, Youth and Sports national research infrastructure Czech-BioImaging projects LM2023050 and CZ.02.1.01/0.0/0.0/18_046/0016045 (M.M., V.U. and M.K.); Czech Science Foundation (GACR) grant GA21-20374S (P.M. and F.L.); European Regional Development Fund in the IT4Innovations national super-computing center–path to exascale project CZ.02.1.01/0.0/0.0/16_013/0001791 within the Operational Programme Research, Development and Education (V.U.); Czech Ministry of Education, Youth and Sports through the e-INFRA CZ project ID:90140 (V.U.); Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación Grant PID2019-109820RB-I00, MCIN/AEI/10.13039/501100011033/, co-financed by the European Regional Development Fund (ERDF), 'A way of making Europe' (P.D.-R., E.G.M., grant to A.M.-B.); BBVA Foundation under a 2017 Leonardo Grant for Researchers and Cultural Creators (A.M.-B.). NVIDIA Corporation for the donation of the Titan X (Pascal) GPU (P.D.-R., E.G.M., to A.M.-B.); the Gulbenkian Foundation, the European Molecular Biology Organization (EMBO) Installation Grant (EMBO-2020-IG-4734) (granted to the Optical Cell Biology laboratory at Instituto Gulbenkian de Ciência) and Postdoctoral Fellowship (EMBO ALTF 174-2022) (E.G.M.); Helmholtz Association program NACIP – Natural, Artificial and Cognitive Information Processing and Biointerfaces International Graduate School (BIF-IGS) (T.S. and R.M.), and HIDSS4Health – Helmholtz Information and Data Science School for Health (K.L. and R.M.); European Research Council, under the European Union Horizon 2020 programme, grant ERC-2015-AdG 694918 (K.S.); Helmholtz Imaging (F.I., P.F.J.); the Negev scholarship at Ben-Gurion University (A.A.); the Kreitman School of Advanced Graduate Studies (A.A.); Israel Ministry of Science, Technology and Space (MOST 3-14344 T.R.R.); the United States – Israel Binational Science Foundation (BSF 2019135 T.R.R.); Human Frontiers Science Program Grant RGP0043/2019 (A.R.C. and L.A.); the Brazilian funding agencies FACEPE, CAPES and CNPq (F.A.G.P., T.I.R.); Beckman Institute at Caltech (A.C., F.A.G.P.); Howard Hughes Medical Institute (E.M.M.); and the USA NIH NINDS R01NS110915 (K.P.) and USA ARL W911NF-18-20285 (K.P.). These authors contributed equally: Michal Kozubek, Carlos Ortiz-de-Solórzano. Contributions. M.M. conceived and designed the analysis, collected the data, contributed data or analysis tools, performed the analysis and wrote the paper; V.U. conceived and designed the analysis, contributed data or analysis tools, and performed the analysis; P.D.-R. and E.G.M. collected the data and performed the analysis; T.N. performed the analysis; F.A.G.P., T.I.R., E.M.M., T.S., K.L., R.M., T.G., Y.W., J.P.A., R.B., N.M.A.-S., G.R., I.E.T., K.P., F.L., P.M., K.S., K.E.G.M., L.A., A.R.C., A.A., T.B.-H., T.R.R., F.I., P.F.J., K.H.M.-H. and Y.Z. contributed data or analysis tools; C.E. and A.U. collected the data, other contribution (annotated data from ground truth); E.M. and A.C. conceived and designed the analysis; and A.M.-B., M.K. and C.O.S. conceived and designed the analysis and wrote the paper. Data availability. The training datasets with their reference annotations and test datasets used in the challenge are publicly available at the CTC website (http://celltrackingchallenge.net). With regard to the raw data for individual Figures and Tables, Source data are provided with this paper. Code availability. The evaluation routines used to produce the results reported in this article are freely available at the CTC website or the CellTrackingChallenge update site as a Fiji plugin, the source codes of which can be found at https://github.com/CellTrackingChallenge/. Furthermore, this public GitHub repository contains links to the executable versions of the individual algorithms and Colab Notebooks of those 11 participants who agreed to share their tools in a reusable form. The parameters used by the participants to produce their benchmarked results are listed on the CTC website. Competing interests. K.S. is employed part-time by LPIXEL Inc. All other authors have no competing interests.

Files

41592_2023_Article_1879.pdf
Files (15.9 MB)
Name Size Download all
md5:66857846de053b27b9d10a6f06a9ff1c
48.2 kB Download
md5:b8e92767f2fe8628ebba6aad6f52008a
44.5 kB Download
md5:2fbd3309267e637e459de12ad8bd9fb6
54.0 kB Download
md5:cb3504c54f32d8b321d723dc9949806e
5.7 MB Preview Download
md5:eb00e845ecd90a41e9903c4f86d63223
39.1 kB Download
md5:6e85cd59ee71d406ae914c98d7abae55
9.8 MB Preview Download
md5:08a4afbbd5aff016ebfb6eb6d615c060
17.7 kB Download
md5:8c014271b8a59e43971a4a9950383466
15.0 kB Download
md5:1116ec2fa08fda0b329e486dd7ea7d47
30.9 kB Download
md5:739a6f87d8de556cae66a564036d0fd5
23.8 kB Download
md5:c40c212eccbe39925fe989b535347be0
38.5 kB Download
md5:b0effa153b6fb082f66f75034141f2ed
42.5 kB Download
md5:89dad153d04f76f2663d83c73b9db36c
53.3 kB Download
md5:4948bd97e11198c4aa9ca86a9a113eb5
38.9 kB Download
md5:6a414ad9fa7a87b0ddc70565d71d9425
16.3 kB Download
md5:82faea63b54781fde312aa928c669f8e
17.8 kB Download

Additional details

Created:
November 22, 2023
Modified:
January 9, 2024