Multi-Object Tracking and Pose Estimation for Animals

The study of collective behaviour among animals hinges on accurately estimating and tracking their movements. Central to our research at the Centre for the Advanced Study of Collective Behaviour is the challenge of accurately estimating the poses and tracking the movements of multiple animals in their natural habitats. One of the primary challenges in studying animal collective behaviour is the scarcity of data, which predominantly focuses on human subjects, with a notable deficiency in annotated data for animals. Although there are some existing animal datasets, they often encompass various species, limiting their utility for specific research purposes. Moreover, the availability of annotated multi-instance animal data remains sparse compared to human datasets, further exacerbating the resource gap. We propose novel approaches to address this disparity and facilitate advancements in pose estimation and tracking methodologies for collective behaviour studies. Firstly, we leverage pigeon data collected in controllable indoor environments to train models capable of performing reliably in wild settings. We also demonstrate that it is possible to train a model on data containing a single pigeon to predict keypoints from multiple pigeons stably and accurately. This provides an alternative in the domain shift to other species. With interactive speed, this model tracks and estimates the 3D poses of up to ten pigeons. Second, we explore unsupervised label propagation that obviates the need for annotated data to propagate poses through video sequences. Our pipeline can effectively track the posture of small objects relative to the frame size, enhancing the applicability. Third, our pioneering 3D pose estimation pipeline, trained exclusively on synthetic data, robustly predicts keypoints from multi-view silhouettes and is thus robust to transformations that leave silhouettes unchanged, such as variations in texture and lighting. This method successfully narrows the domain gap where real-world annotations are scarce by leveraging synthetic data. Lastly, to the best of our knowledge, we are the first to offer a pipeline for neural rendering of textures, facilitating downstream tasks such as individual re-identification. Our method offers an efficient alternative to existing approaches based on convolutional neural networks (CNNs) and vision transformers, operating at interactive speeds. We think that our contributions promote systematic advancements in the study of animal collective behaviour and offer novel methodologies for 3D pose estimation and individual re-identification.

3D-MuPPET: 3D Multi-Pigeon Pose Estimation and Tracking

Project page for '3D-MuPPET: 3D Multi-Pigeon Pose Estimation and Tracking'

WACV 2024 Open Access Repository

3D-MuPPET: 3D Multi-Pigeon Pose Estimation and Tracking

Project page for '3D-MuPPET: 3D Multi-Pigeon Pose Estimation and Tracking'

I-MuPPET: Interactive Multi-Pigeon Pose Estimation and Tracking (Dataset)

This data entry contains the annotated single pigeon data from the I-MuPPET GCPR 2022 paper. This data contains our annotated single pigeon data with RGB images and seven distinct keypoint annotations. The experiments were carried out by Hemal Naik, Máté Nagy, Fumihiro Kano and Iain D. Couzin and were approved by the Regierungspräsidium Freiburg under the permit number Az. 35-9185.81/G-19/107. Data was recorded by two Vicon Vue 2 cameras (1920x1080 pixels) at 50 Hz, four Vantage 5 and 26 Vero 2.2 sensors. The dataset was originally created during the Ph.D. thesis work of Hemal Naik. The method to reproduce the setup and dataset can be found at https://mediatum.ub.tum.de/?id=1554403. An updated version of this dataset with 4K resolution (3D-POP) is available at https://doi.org/10.17617/3.HPBBC7. The code base is available at https://github.com/alexhang212/dataset-3dpop. The users can reproduce annotations or improve them by adding more key points. Code for I-MuPPET available at https://github.com/urs-waldmann/i-muppet/.

I-MuPPET: Interactive Multi-Pigeon Pose Estimation and Tracking (Tracking Benchmark)

This data entry contains the multi-pigeon video sequences with ground truth for the quantitative tracking evaluation from the I-MuPPET GCPR 2022 paper. This data contains 24 video sequences showing up to four pigeons in our imaging barn at the Centre for the Advanced Study of Collective Behaviour. The experiments were carried out by Hemal Naik, Máté Nagy, Fumihiro Kano and Iain D. Couzin and were approved by the Regierungspräsidium Freiburg under the permit number Az. 35-9185.81/G-19/107. Data was recorded by two Vicon Vue 2 cameras (1920x1080 pixels) at 50 Hz. The ground truth for the quantitative tracking evaluation was created by Urs Waldmann. Code for I-MuPPET available at https://github.com/urs-waldmann/i-muppet/.

I-MuPPET: Interactive Multi-Pigeon Pose Estimation and Tracking (Weights for Pigeons)

This data entry contains pre-trained weights from the I-MuPPET GCPR 2022 paper. This data contains pre-trained weights for pigeons. The weights were trained with I-MuPPET by Urs Waldmann on our labeled single pigeon data set. Code for I-MuPPET available at https://github.com/urs-waldmann/i-muppet/.

Neural Puppeteer: Keypoint-Based Neural Rendering of Dynamic Shapes

We introduce Neural Puppeteer, an efficient neural rendering pipeline for articulated shapes. By inverse rendering, we can predict 3D keypoints from multi-view 2D silhouettes alone, without requiring texture information. Furthermore, we can easily predict 3D...