| website | https://hermannblum.net |
It takes a while to make fancy #NeRF animations, so I am very happy we can now share our upcoming #CVPR paper with video and code release:
A big debate in #ContinualLearning is how to scale to many experiences. This work shows how well NeRF-based compression can scale to store robotic experiences over many consecutive deployments, much better than storing checkpoints of your model.
website: https://ethz-asl.github.io/ucsa_neural_rendering/
#Robotics #SceneUnderstanding
Semantic segmentation networks are usually pre-trained once and not updated during deployment. As a consequence, misclassifications commonly occur if the distribution of the training data deviates from the one encountered during the robot's operation. We propose to mitigate this problem by adapting the neural network to the robot's environment during deployment, without any need for external supervision. Leveraging complementary data representations, we generate a supervision signal, by probabilistically accumulating consecutive 2D semantic predictions in a volumetric 3D map. We then train the network on renderings of the accumulated semantic map, effectively resolving ambiguities and enforcing multi-view consistency through the 3D representation. In contrast to scene adaptation methods, we aim to retain the previously-learned knowledge, and therefore employ a continual learning experience replay strategy to adapt the network. Through extensive experimental evaluation, we show successful adaptation to real-world indoor scenes both on the ScanNet dataset and on in-house data recorded with an RGB-D sensor. Our method increases the segmentation accuracy on average by 9.9% compared to the fixed pre-trained neural network, while retaining knowledge from the pre-training dataset.