Video object segmentation using teacher-student adaptation in a human robot interaction (HRI) setting

Mennatullah Siam, Chen Jiang, Steven Lu, Laura Petrich, Mahmoud Gamal, Mohamed Elhoseiny, Martin Jagersand

Research output: Chapter in Book/Report/Conference proceedingConference contribution

61 Scopus citations


Video object segmentation is an essential task in robot manipulation to facilitate grasping and learning affordances. Incremental learning is important for robotics in unstructured environments. Inspired by the children learning process, human robot interaction (HRI) can be utilized to teach robots about the world guided by humans similar to how children learn from a parent or a teacher. A human teacher can show potential objects of interest to the robot, which is able to self adapt to the teaching signal without providing manual segmentation labels. We propose a novel teacher-student learning paradigm to teach robots about their surrounding environment. A two-stream motion and appearance 'teacher' network provides pseudo-labels to adapt an appearance 'student' network. The student network is able to segment the newly learned objects in other scenes, whether they are static or in motion. We also introduce a carefully designed dataset that serves the proposed HRI setup, denoted as (I)nteractive (V)ideo (O)bject (S)egmentation. Our IVOS dataset contains teaching videos of different objects, and manipulation tasks. Our proposed adaptation method outperforms the state-of-theart on DAVIS and FBMS with 6.8% and 1.2% in F-measure respectively. It improves over the baseline on IVOS dataset with 46.1% and 25.9% in mIoU.
Original languageEnglish (US)
Title of host publicationProceedings - IEEE International Conference on Robotics and Automation
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Print)9781538660263
StatePublished - May 1 2019
Externally publishedYes

Bibliographical note

Generated from Scopus record by KAUST IRTS on 2019-11-20


Dive into the research topics of 'Video object segmentation using teacher-student adaptation in a human robot interaction (HRI) setting'. Together they form a unique fingerprint.

Cite this