Audio-visual Speaker Diarization: Improved Voice Activity Detection with CNN based Feature Extraction

Konstantinos Fanaras; Antonios Tragoudaras; Charalampos Antoniadis; Yehia Massoud

doi:10.1109/MWSCAS54063.2022.9859533

Audio-visual Speaker Diarization: Improved Voice Activity Detection with CNN based Feature Extraction

Konstantinos Fanaras, Antonios Tragoudaras, Charalampos Antoniadis, Yehia Massoud

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Scopus citations

Abstract

Speaker diarization is a task to identify 'who spoke when'. Moreover, nowadays, speakers' audio clips usually are accompanied by visual information. Thus, in the latest works, speaker diarization systems performance has been improved substantially by taking advantage of the visual information synchronized with audio clips in Audio-Visual (AV) content. This paper presents a deep learning architecture to implement an AV speaker diarization system emphasizing Voice Activity Detection (VAD). Traditional AV speaker diarization systems use hand-crafted features, like Mel-frequency cepstral coefficients, to perform VAD. On the other hand, the VAD module in our proposed system employs Convolutional Neural Networks (CNN) to learn and extract features from the audio waveforms directly. Experimental results on the AMI Meeting Corpus indicated that the proposed multimodal speaker diarization system reaches a state-of-the-art VAD False Alarm rate due to the CNN-based VAD, which in turn boosts the whole system's performance.

Original language	English (US)
Title of host publication	MWSCAS 2022 - 65th IEEE International Midwest Symposium on Circuits and Systems, Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781665402798
DOIs	https://doi.org/10.1109/MWSCAS54063.2022.9859533
State	Published - 2022
Event	65th IEEE International Midwest Symposium on Circuits and Systems, MWSCAS 2022 - Fukuoka, Japan Duration: Aug 7 2022 → Aug 10 2022

Publication series

Name	Midwest Symposium on Circuits and Systems
Volume	2022-August
ISSN (Print)	1548-3746

Conference

Conference	65th IEEE International Midwest Symposium on Circuits and Systems, MWSCAS 2022
Country/Territory	Japan
City	Fukuoka
Period	08/7/22 → 08/10/22

Bibliographical note

Publisher Copyright:
© 2022 IEEE.

Keywords

audio-visual
convolutional neural networks
deep learning
false alarm rate
speaker diarization
voice activity detection

ASJC Scopus subject areas

Electronic, Optical and Magnetic Materials
Electrical and Electronic Engineering

Access to Document

10.1109/MWSCAS54063.2022.9859533

Cite this

Fanaras, K., Tragoudaras, A., Antoniadis, C., & Massoud, Y. (2022). Audio-visual Speaker Diarization: Improved Voice Activity Detection with CNN based Feature Extraction. In MWSCAS 2022 - 65th IEEE International Midwest Symposium on Circuits and Systems, Proceedings (Midwest Symposium on Circuits and Systems; Vol. 2022-August). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/MWSCAS54063.2022.9859533

Fanaras, Konstantinos ; Tragoudaras, Antonios ; Antoniadis, Charalampos et al. / Audio-visual Speaker Diarization : Improved Voice Activity Detection with CNN based Feature Extraction. MWSCAS 2022 - 65th IEEE International Midwest Symposium on Circuits and Systems, Proceedings. Institute of Electrical and Electronics Engineers Inc., 2022. (Midwest Symposium on Circuits and Systems).

@inproceedings{c9583899b88444f3b52ec4a850fa0040,

title = "Audio-visual Speaker Diarization: Improved Voice Activity Detection with CNN based Feature Extraction",

abstract = "Speaker diarization is a task to identify 'who spoke when'. Moreover, nowadays, speakers' audio clips usually are accompanied by visual information. Thus, in the latest works, speaker diarization systems performance has been improved substantially by taking advantage of the visual information synchronized with audio clips in Audio-Visual (AV) content. This paper presents a deep learning architecture to implement an AV speaker diarization system emphasizing Voice Activity Detection (VAD). Traditional AV speaker diarization systems use hand-crafted features, like Mel-frequency cepstral coefficients, to perform VAD. On the other hand, the VAD module in our proposed system employs Convolutional Neural Networks (CNN) to learn and extract features from the audio waveforms directly. Experimental results on the AMI Meeting Corpus indicated that the proposed multimodal speaker diarization system reaches a state-of-the-art VAD False Alarm rate due to the CNN-based VAD, which in turn boosts the whole system's performance.",

keywords = "audio-visual, convolutional neural networks, deep learning, false alarm rate, speaker diarization, voice activity detection",

author = "Konstantinos Fanaras and Antonios Tragoudaras and Charalampos Antoniadis and Yehia Massoud",

note = "Publisher Copyright: {\textcopyright} 2022 IEEE.; 65th IEEE International Midwest Symposium on Circuits and Systems, MWSCAS 2022 ; Conference date: 07-08-2022 Through 10-08-2022",

year = "2022",

doi = "10.1109/MWSCAS54063.2022.9859533",

language = "English (US)",

series = "Midwest Symposium on Circuits and Systems",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "MWSCAS 2022 - 65th IEEE International Midwest Symposium on Circuits and Systems, Proceedings",

address = "United States",

}

Fanaras, K, Tragoudaras, A, Antoniadis, C & Massoud, Y 2022, Audio-visual Speaker Diarization: Improved Voice Activity Detection with CNN based Feature Extraction. in MWSCAS 2022 - 65th IEEE International Midwest Symposium on Circuits and Systems, Proceedings. Midwest Symposium on Circuits and Systems, vol. 2022-August, Institute of Electrical and Electronics Engineers Inc., 65th IEEE International Midwest Symposium on Circuits and Systems, MWSCAS 2022, Fukuoka, Japan, 08/7/22. https://doi.org/10.1109/MWSCAS54063.2022.9859533

Audio-visual Speaker Diarization: Improved Voice Activity Detection with CNN based Feature Extraction. / Fanaras, Konstantinos; Tragoudaras, Antonios; Antoniadis, Charalampos et al.
MWSCAS 2022 - 65th IEEE International Midwest Symposium on Circuits and Systems, Proceedings. Institute of Electrical and Electronics Engineers Inc., 2022. (Midwest Symposium on Circuits and Systems; Vol. 2022-August).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Audio-visual Speaker Diarization

T2 - 65th IEEE International Midwest Symposium on Circuits and Systems, MWSCAS 2022

AU - Fanaras, Konstantinos

AU - Tragoudaras, Antonios

AU - Antoniadis, Charalampos

AU - Massoud, Yehia

PY - 2022

Y1 - 2022

N2 - Speaker diarization is a task to identify 'who spoke when'. Moreover, nowadays, speakers' audio clips usually are accompanied by visual information. Thus, in the latest works, speaker diarization systems performance has been improved substantially by taking advantage of the visual information synchronized with audio clips in Audio-Visual (AV) content. This paper presents a deep learning architecture to implement an AV speaker diarization system emphasizing Voice Activity Detection (VAD). Traditional AV speaker diarization systems use hand-crafted features, like Mel-frequency cepstral coefficients, to perform VAD. On the other hand, the VAD module in our proposed system employs Convolutional Neural Networks (CNN) to learn and extract features from the audio waveforms directly. Experimental results on the AMI Meeting Corpus indicated that the proposed multimodal speaker diarization system reaches a state-of-the-art VAD False Alarm rate due to the CNN-based VAD, which in turn boosts the whole system's performance.

AB - Speaker diarization is a task to identify 'who spoke when'. Moreover, nowadays, speakers' audio clips usually are accompanied by visual information. Thus, in the latest works, speaker diarization systems performance has been improved substantially by taking advantage of the visual information synchronized with audio clips in Audio-Visual (AV) content. This paper presents a deep learning architecture to implement an AV speaker diarization system emphasizing Voice Activity Detection (VAD). Traditional AV speaker diarization systems use hand-crafted features, like Mel-frequency cepstral coefficients, to perform VAD. On the other hand, the VAD module in our proposed system employs Convolutional Neural Networks (CNN) to learn and extract features from the audio waveforms directly. Experimental results on the AMI Meeting Corpus indicated that the proposed multimodal speaker diarization system reaches a state-of-the-art VAD False Alarm rate due to the CNN-based VAD, which in turn boosts the whole system's performance.

KW - audio-visual

KW - convolutional neural networks

KW - deep learning

KW - false alarm rate

KW - speaker diarization

KW - voice activity detection

UR - http://www.scopus.com/inward/record.url?scp=85137424359&partnerID=8YFLogxK

U2 - 10.1109/MWSCAS54063.2022.9859533

DO - 10.1109/MWSCAS54063.2022.9859533

M3 - Conference contribution

AN - SCOPUS:85137424359

T3 - Midwest Symposium on Circuits and Systems

BT - MWSCAS 2022 - 65th IEEE International Midwest Symposium on Circuits and Systems, Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 7 August 2022 through 10 August 2022

ER -

Fanaras K, Tragoudaras A, Antoniadis C, Massoud Y. Audio-visual Speaker Diarization: Improved Voice Activity Detection with CNN based Feature Extraction. In MWSCAS 2022 - 65th IEEE International Midwest Symposium on Circuits and Systems, Proceedings. Institute of Electrical and Electronics Engineers Inc. 2022. (Midwest Symposium on Circuits and Systems). doi: 10.1109/MWSCAS54063.2022.9859533

Audio-visual Speaker Diarization: Improved Voice Activity Detection with CNN based Feature Extraction

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this