Automated audiovisual behavior recognition in wild primates
Abstract
Large video datasets of wild animal behavior are crucial to produce longitudinal research and accelerate conservation efforts; however, large-scale behavior analyses continue to be severely constrained by time and resources. We present a deep convolutional neural network approach and fully automated pipeline to detect and track two audiovisually distinctive actions in wild chimpanzees: buttress drumming and nut cracking. Using camera trap and direct video recordings, we train action recognition models using audio and visual signatures of both behaviors, attaining high average precision (buttress drumming: 0.87 and nut cracking: 0.85), and demonstrate the potential for behavioral analysis using the automatically parsed video. Our approach produces the first automated audiovisual action recognition of wild primate behavior, setting a milestone for exploiting large datasets in ethology and conservation.
Additional Information
© 2021 The Authors, some rights reserved; exclusive licensee American Association for the Advancement of Science. No claim to original U.S. Government Works. Distributed under a Creative Commons Attribution NonCommercial License 4.0 (CC BY-NC). Submitted 12 March 2021; Accepted 23 September 2021; Published 12 November 2021. We are grateful to Kyoto University's Primate Research Institute for leading the Bossou Archive Project and supporting the research presented here and to the IREB and DNRSIT of Guinea. This study is dedicated to all the researchers and field assistants who have collected data in Bossou since 1988. We thank the Instituto da Biodiversidade e das Áreas Protegidas (IBAP) for their permission to conduct research in Guinea-Bissau and for logistical support, research assistants and local guides for assisting with data collection, and local leaders for granting us permission to conduct research. We thank M. Ramon for collecting camera trap data in Cabante, Guinea-Bissau. This study was supported by EPSRC Programme Grants Seebibyte EP/M013774/1 and Visual AI EP/T028572/1; Google PhD Fellowship (to A.N.); Clarendon Fund (to D.S. and S.B.); Boise Trust Fund (to D.S., S.B., and J.B.); Wolfson College, University of Oxford (to D.S.); Keble College Sloane-Robinson Clarendon Scholarship, University of Oxford (to S.B.); Fundação para a Ciência e a Tecnologia, Portugal SFRH/BD/108185/2015 (to J.B.); Templeton World Charity Foundation grant no. TWCF0316 (to D.B.); National Geographic Society (to S.C.); St Hugh's College, University of Oxford (to S.C.); Kyoto University Primate Research Institute for Cooperative Research Program (to M.H. and D.S.); MEXT-JSPS (no. 16H06283), LGP-U04, the Japan Society for the Promotion of Science (to T.M.); and Darwin Initiative funding grant number 26-018 (to K.J.H.). Author contributions: Conceptualization: D.S. Methodology: M.B., A.N., and A.Z. Data curation: M.B., D.S., J.B., S.B., and J.O. Data collection: D.B., S.C., T.M., M.H., K.J.H., and J.B. Software, formal analysis, and visualization: M.B. Supervision: A.Z., D.B., and S.C. Writing (original draft): M.B., A.N., D.S., and J.B. Writing (review and editing): A.Z., D.B., S.C., and K.J.H. The authors declare that they have no competing interests. Data and materials availability: All data needed to evaluate the conclusions in the paper are present in the paper and/or the Supplementary Materials and at the Dryad data repository (https://datadryad.org/stash/share/UUfSTzsL9eTbAo-78pdaXPdaIUJmdJzSuqhXcb48vHM).Attached Files
Published - sciadv.abi4883.pdf
Supplemental Material - sciadv.abi4883_movie_s1.zip
Supplemental Material - sciadv.abi4883_sm.pdf
Files
Additional details
- PMCID
- PMC8589313
- Eprint ID
- 111868
- Resolver ID
- CaltechAUTHORS:20211115-173820409
- EP/M013774/1
- Engineering and Physical Sciences Research Council (EPSRC)
- EP/T028572/1
- Engineering and Physical Sciences Research Council (EPSRC)
- Google PhD Fellowship
- Clarendon Fund
- Boise Trust Fund
- Wolfson College, Oxford
- Keble College Sloane-Robinson Clarendon Scholarship, Oxford
- SFRH/BD/108185/2015
- Fundação para a Ciência e a Tecnologia (FCT)
- TWCF0316
- Templeton World Charity Foundation
- National Geographic Society
- St Hugh's College, Oxford
- Kyoto University Primate Research Institute
- 16H06283
- Ministry of Education, Culture, Sports, Science and Technology (MEXT)
- LGP-U04
- Japan Society for the Promotion of Science (JSPS)
- 26-018
- Darwin Initiative
- Created
-
2021-11-15Created from EPrint's datestamp field
- Updated
-
2021-11-19Created from EPrint's last_modified field