Investigations on End- to-End Audiovisual Fusion

Michael Wand; Jurgen Schmidhuber; Ngoc Thang Vu

doi:10.1109/ICASSP.2018.8461900

Investigations on End- to-End Audiovisual Fusion

Michael Wand, Jurgen Schmidhuber, Ngoc Thang Vu

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

25 Scopus citations

Abstract

Audiovisual speech recognition (AVSR) is a method to alleviate the adverse effect of noise in the acoustic signal. Leveraging recent developments in deep neural network-based speech recognition, we present an AVSR neural network architecture which is trained end-to-end, without the need to separately model the process of decision fusion as in conventional (e.g. HMM-based) systems. The fusion system outperforms single-modality recognition under all noise conditions. Investigation of the saliency of the input features shows that the neural network automatically adapts to different noise levels in the acoustic signal.

Original language	English (US)
Title of host publication	ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	3041-3045
Number of pages	5
ISBN (Print)	9781538646588
DOIs	https://doi.org/10.1109/ICASSP.2018.8461900
State	Published - Sep 10 2018
Externally published	Yes

Bibliographical note

Generated from Scopus record by KAUST IRTS on 2022-09-14

Access to Document

10.1109/ICASSP.2018.8461900

Cite this

@inproceedings{c6bc51023b50463984ff4282e7318bf4,

title = "Investigations on End- to-End Audiovisual Fusion",

abstract = "Audiovisual speech recognition (AVSR) is a method to alleviate the adverse effect of noise in the acoustic signal. Leveraging recent developments in deep neural network-based speech recognition, we present an AVSR neural network architecture which is trained end-to-end, without the need to separately model the process of decision fusion as in conventional (e.g. HMM-based) systems. The fusion system outperforms single-modality recognition under all noise conditions. Investigation of the saliency of the input features shows that the neural network automatically adapts to different noise levels in the acoustic signal.",

author = "Michael Wand and Jurgen Schmidhuber and Vu, {Ngoc Thang}",

note = "Generated from Scopus record by KAUST IRTS on 2022-09-14",

year = "2018",

month = sep,

day = "10",

doi = "10.1109/ICASSP.2018.8461900",

language = "English (US)",

isbn = "9781538646588",

pages = "3041--3045",

booktitle = "ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

address = "United States",

}

TY - GEN

T1 - Investigations on End- to-End Audiovisual Fusion

AU - Wand, Michael

AU - Schmidhuber, Jurgen

AU - Vu, Ngoc Thang

N1 - Generated from Scopus record by KAUST IRTS on 2022-09-14

PY - 2018/9/10

Y1 - 2018/9/10

N2 - Audiovisual speech recognition (AVSR) is a method to alleviate the adverse effect of noise in the acoustic signal. Leveraging recent developments in deep neural network-based speech recognition, we present an AVSR neural network architecture which is trained end-to-end, without the need to separately model the process of decision fusion as in conventional (e.g. HMM-based) systems. The fusion system outperforms single-modality recognition under all noise conditions. Investigation of the saliency of the input features shows that the neural network automatically adapts to different noise levels in the acoustic signal.

AB - Audiovisual speech recognition (AVSR) is a method to alleviate the adverse effect of noise in the acoustic signal. Leveraging recent developments in deep neural network-based speech recognition, we present an AVSR neural network architecture which is trained end-to-end, without the need to separately model the process of decision fusion as in conventional (e.g. HMM-based) systems. The fusion system outperforms single-modality recognition under all noise conditions. Investigation of the saliency of the input features shows that the neural network automatically adapts to different noise levels in the acoustic signal.

UR - https://ieeexplore.ieee.org/document/8461900/

UR - http://www.scopus.com/inward/record.url?scp=85051681116&partnerID=8YFLogxK

U2 - 10.1109/ICASSP.2018.8461900

DO - 10.1109/ICASSP.2018.8461900

M3 - Conference contribution

SN - 9781538646588

SP - 3041

EP - 3045

BT - ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Investigations on End- to-End Audiovisual Fusion

Abstract

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this