MAAS: Multi-modal Assignation for Active Speaker Detection

Juan León Alcázar; Fabian Caba Heilbron; Ali K. Thabet; Bernard Ghanem

doi:10.1109/ICCV48922.2021.00033

MAAS: Multi-modal Assignation for Active Speaker Detection

Juan León Alcázar, Fabian Caba Heilbron, Ali K. Thabet, Bernard Ghanem

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

27 Scopus citations

Abstract

Active speaker detection requires a mindful integration of multi-modal cues. Current methods focus on modeling and fusing short-term audiovisual features for individual speakers, often at frame level. We present a novel approach to active speaker detection that directly addresses the multi-modal nature of the problem and provides a straightforward strategy, where independent visual features (speakers) in the scene are assigned to a previously detected speech event. Our experiments show that a small graph data structure built from local information can approximate an instantaneous audio-visual assignment problem. Moreover, the temporal extension of this initial graph achieves a new state-of-the-art performance on the AVA-ActiveSpeaker dataset with a mAP of 88.8%.

Original language	English (US)
Title of host publication	Proceedings - 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	265-274
Number of pages	10
ISBN (Electronic)	9781665428125
DOIs	https://doi.org/10.1109/ICCV48922.2021.00033
State	Published - 2021
Event	18th IEEE/CVF International Conference on Computer Vision, ICCV 2021 - Virtual, Online, Canada Duration: Oct 11 2021 → Oct 17 2021

Publication series

Name	Proceedings of the IEEE International Conference on Computer Vision
ISSN (Print)	1550-5499

Conference

Conference	18th IEEE/CVF International Conference on Computer Vision, ICCV 2021
Country/Territory	Canada
City	Virtual, Online
Period	10/11/21 → 10/17/21

Bibliographical note

Funding Information:
Acknowledgments. This work was supported by the King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research through the Visual Computing Center (VCC) funding.

Publisher Copyright:
© 2021 IEEE

ASJC Scopus subject areas

Software
Computer Vision and Pattern Recognition

Access to Document

10.1109/ICCV48922.2021.00033

Cite this

Alcázar, J. L., Heilbron, F. C., Thabet, A. K., & Ghanem, B. (2021). MAAS: Multi-modal Assignation for Active Speaker Detection. In Proceedings - 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021 (pp. 265-274). (Proceedings of the IEEE International Conference on Computer Vision). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICCV48922.2021.00033

@inproceedings{1049419a2c2040558eea92be5853f8a1,

title = "MAAS: Multi-modal Assignation for Active Speaker Detection",

abstract = "Active speaker detection requires a mindful integration of multi-modal cues. Current methods focus on modeling and fusing short-term audiovisual features for individual speakers, often at frame level. We present a novel approach to active speaker detection that directly addresses the multi-modal nature of the problem and provides a straightforward strategy, where independent visual features (speakers) in the scene are assigned to a previously detected speech event. Our experiments show that a small graph data structure built from local information can approximate an instantaneous audio-visual assignment problem. Moreover, the temporal extension of this initial graph achieves a new state-of-the-art performance on the AVA-ActiveSpeaker dataset with a mAP of 88.8%.",

author = "Alc{\'a}zar, {Juan Le{\'o}n} and Heilbron, {Fabian Caba} and Thabet, {Ali K.} and Bernard Ghanem",

note = "Funding Information: Acknowledgments. This work was supported by the King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research through the Visual Computing Center (VCC) funding. Publisher Copyright: {\textcopyright} 2021 IEEE; 18th IEEE/CVF International Conference on Computer Vision, ICCV 2021 ; Conference date: 11-10-2021 Through 17-10-2021",

year = "2021",

doi = "10.1109/ICCV48922.2021.00033",

language = "English (US)",

series = "Proceedings of the IEEE International Conference on Computer Vision",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "265--274",

booktitle = "Proceedings - 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021",

address = "United States",

}

Alcázar, JL, Heilbron, FC, Thabet, AK & Ghanem, B 2021, MAAS: Multi-modal Assignation for Active Speaker Detection. in Proceedings - 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021. Proceedings of the IEEE International Conference on Computer Vision, Institute of Electrical and Electronics Engineers Inc., pp. 265-274, 18th IEEE/CVF International Conference on Computer Vision, ICCV 2021, Virtual, Online, Canada, 10/11/21. https://doi.org/10.1109/ICCV48922.2021.00033

MAAS: Multi-modal Assignation for Active Speaker Detection. / Alcázar, Juan León; Heilbron, Fabian Caba; Thabet, Ali K. et al.
Proceedings - 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021. Institute of Electrical and Electronics Engineers Inc., 2021. p. 265-274 (Proceedings of the IEEE International Conference on Computer Vision).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - MAAS

T2 - 18th IEEE/CVF International Conference on Computer Vision, ICCV 2021

AU - Alcázar, Juan León

AU - Heilbron, Fabian Caba

AU - Thabet, Ali K.

AU - Ghanem, Bernard

N1 - Funding Information: Acknowledgments. This work was supported by the King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research through the Visual Computing Center (VCC) funding. Publisher Copyright: © 2021 IEEE

PY - 2021

Y1 - 2021

N2 - Active speaker detection requires a mindful integration of multi-modal cues. Current methods focus on modeling and fusing short-term audiovisual features for individual speakers, often at frame level. We present a novel approach to active speaker detection that directly addresses the multi-modal nature of the problem and provides a straightforward strategy, where independent visual features (speakers) in the scene are assigned to a previously detected speech event. Our experiments show that a small graph data structure built from local information can approximate an instantaneous audio-visual assignment problem. Moreover, the temporal extension of this initial graph achieves a new state-of-the-art performance on the AVA-ActiveSpeaker dataset with a mAP of 88.8%.

AB - Active speaker detection requires a mindful integration of multi-modal cues. Current methods focus on modeling and fusing short-term audiovisual features for individual speakers, often at frame level. We present a novel approach to active speaker detection that directly addresses the multi-modal nature of the problem and provides a straightforward strategy, where independent visual features (speakers) in the scene are assigned to a previously detected speech event. Our experiments show that a small graph data structure built from local information can approximate an instantaneous audio-visual assignment problem. Moreover, the temporal extension of this initial graph achieves a new state-of-the-art performance on the AVA-ActiveSpeaker dataset with a mAP of 88.8%.

UR - http://www.scopus.com/inward/record.url?scp=85125026260&partnerID=8YFLogxK

U2 - 10.1109/ICCV48922.2021.00033

DO - 10.1109/ICCV48922.2021.00033

M3 - Conference contribution

AN - SCOPUS:85125026260

T3 - Proceedings of the IEEE International Conference on Computer Vision

SP - 265

EP - 274

BT - Proceedings - 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 11 October 2021 through 17 October 2021

ER -

Alcázar JL, Heilbron FC, Thabet AK, Ghanem B. MAAS: Multi-modal Assignation for Active Speaker Detection. In Proceedings - 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021. Institute of Electrical and Electronics Engineers Inc. 2021. p. 265-274. (Proceedings of the IEEE International Conference on Computer Vision). doi: 10.1109/ICCV48922.2021.00033

MAAS: Multi-modal Assignation for Active Speaker Detection

Abstract

Publication series

Conference

Bibliographical note

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this