Jens Rahnfeld

Project Page: https://github.com/aselimc/Clustering-Segmentation-Based-On-iBOT

Motivation

Collecting lots of data or more specifically, images is often very easy. However annotating them by a human expert can be very time consuming and even infeasible due to limited budget. So this project focuses on the situation where we have a large dataset with only little annotated data for semantic segmentation. Furthermore, training to learn feature representations from scratch for this new dataset would be computationally expensive. Therefore we use iBOT, which is a self-supervised method that is pretrained on ImageNet, to obtain feature space representations of images. By applying clustering methods on top of iBOT features we then obtain semantic segmentation maps on PASCAL VOC.

Abstract

Predicting the future is a key ability for robots as it enables them to reason about actions that are beneficial and avoid dangerous real-world situations. Existing methods for video prediction operate only on RGB frames. However, robots feature not only RGB sensors, but also depth sensors. To this end, we propose a novel approach to action-conditioned video prediction that utilizes additional spatial information from depth maps to reason about 3D motion. We compare our proposed method against existing state-of-the-art methods on a synthetic dataset of simulated object manipulations by a Shadow Hand robot. Quantitative evaluations demonstrate that we achieve competitive results in terms of standard image quality metrics in the action-conditioned setting.