SuperAnimal pretrained pose estimation models for behavioral analysis

Shaokai Ye, Anastasiia Filippova, Jessy Lauer, Steffen Schneider, Maxime Vidal, Tian Qiu, Alexander Mathis and Mackenzie Weygandt Mathis ()
Additional contact information
Shaokai Ye: Brain Mind Institute & Neuro-X Institute
Anastasiia Filippova: Brain Mind Institute & Neuro-X Institute
Jessy Lauer: Brain Mind Institute & Neuro-X Institute
Steffen Schneider: Brain Mind Institute & Neuro-X Institute
Maxime Vidal: Brain Mind Institute & Neuro-X Institute
Tian Qiu: Brain Mind Institute & Neuro-X Institute
Alexander Mathis: Brain Mind Institute & Neuro-X Institute
Mackenzie Weygandt Mathis: Brain Mind Institute & Neuro-X Institute

Nature Communications, 2024, vol. 15, issue 1, 1-19

Abstract: Abstract Quantification of behavior is critical in diverse applications from neuroscience, veterinary medicine to animal conservation. A common key step for behavioral analysis is first extracting relevant keypoints on animals, known as pose estimation. However, reliable inference of poses currently requires domain knowledge and manual labeling effort to build supervised models. We present SuperAnimal, a method to develop unified foundation models that can be used on over 45 species, without additional manual labels. These models show excellent performance across six pose estimation benchmarks. We demonstrate how to fine-tune the models (if needed) on differently labeled data and provide tooling for unsupervised video adaptation to boost performance and decrease jitter across frames. If fine-tuned, SuperAnimal models are 10–100× more data efficient than prior transfer-learning-based approaches. We illustrate the utility of our models in behavioral classification and kinematic analysis. Collectively, we present a data-efficient solution for animal pose estimation.

Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.nature.com/articles/s41467-024-48792-2 Abstract (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-48792-2

Ordering information: This journal article can be ordered from
https://www.nature.com/ncomms/

DOI: 10.1038/s41467-024-48792-2

Access Statistics for this article

Nature Communications is currently edited by Nathalie Le Bot, Enda Bergin and Fiona Gillespie

More articles in Nature Communications from Nature
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().