MAIN: Multi-Attention Instance Network for video segmentation

Juan León Alcázar, María A. Bravo, Guillaume Jeanneret, Ali Kassem Thabet, Thomas Brox, Pablo Arbeláez, Bernard Ghanem

Research output: Contribution to journalArticlepeer-review


Instance-level video segmentation requires a solid integration of spatial and temporal information. However, current methods rely mostly on domain-specific information (online learning) to produce accurate instance-level segmentations. We propose a novel approach that relies exclusively on the integration of generic spatio-temporal attention cues. Our strategy, named Multi-Attention Instance Network (MAIN), overcomes challenging segmentation scenarios over arbitrary videos without modelling sequence- or instance-specific knowledge. We design MAIN to segment multiple instances in a single forward pass, and optimize it with a novel loss function that favors class agnostic predictions and assigns instance-specific penalties. We achieve state-of-the-art performance on the challenging Youtube-VOS dataset and benchmark, improving the unseen Jaccard and F-Metric by 6.8% and 12.7% respectively, while operating at real-time (30.3 FPS).
Original languageEnglish (US)
Pages (from-to)103240
JournalComputer Vision and Image Understanding
StatePublished - Jun 24 2021

Bibliographical note

KAUST Repository Item: Exported on 2021-06-28
Acknowledgements: This work was partially supported by the King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research, and by the German-Colombian Academic Cooperation between the German Research Foundation (DFG grant BR 3815/9-1) and Universidad de los Andes , Colombia.

ASJC Scopus subject areas

  • Signal Processing
  • Software
  • Computer Vision and Pattern Recognition


Dive into the research topics of 'MAIN: Multi-Attention Instance Network for video segmentation'. Together they form a unique fingerprint.

Cite this