Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition

Srijith Radhakrishnan; Chao Han Huck Yang; Sumeer Ahmad Khan; Rohit Kumar; Narsis A. Kiani; David Gomez-Cabrero; Jesper N. Tegner

Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition

Srijith Radhakrishnan, Chao Han Huck Yang, Sumeer Ahmad Khan, Rohit Kumar, Narsis A. Kiani, David Gomez-Cabrero, Jesper N. Tegner

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

14 Scopus citations

Abstract

We introduce a new cross-modal fusion technique designed for generative error correction in automatic speech recognition (ASR). Our methodology leverages both acoustic information and external linguistic representations to generate accurate speech transcription contexts. This marks a step towards a fresh paradigm in generative error correction within the realm of n-best hypotheses. Unlike the existing ranking-based rescoring methods, our approach adeptly uses distinct initialization techniques and parameter-efficient algorithms to boost ASR performance derived from pre-trained speech and text models. Through evaluation across diverse ASR datasets, we assess our fusion technique, demonstrating a 37.66% improvement in word error rate (WER) relative performance compared to the n-best Oracle. To encourage future research, we have made our code and pre-trained models open source at https://github.com/Srijith-rkr/Whispering-LLaMA.

Original language	English (US)
Title of host publication	EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings
Editors	Houda Bouamor, Juan Pino, Kalika Bali
Publisher	Association for Computational Linguistics (ACL)
Pages	10007-10016
Number of pages	10
ISBN (Electronic)	9798891760608
State	Published - 2023
Event	2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 - Hybrid, Singapore, Singapore Duration: Dec 6 2023 → Dec 10 2023

Publication series

Name	EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings

Conference

Conference	2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023
Country/Territory	Singapore
City	Hybrid, Singapore
Period	12/6/23 → 12/10/23

Bibliographical note

Publisher Copyright:
© 2023 Association for Computational Linguistics.

ASJC Scopus subject areas

Computational Theory and Mathematics
Computer Science Applications
Information Systems
Linguistics and Language

Cite this

Radhakrishnan, S., Yang, C. H. H., Khan, S. A., Kumar, R., Kiani, N. A., Gomez-Cabrero, D., & Tegner, J. N. (2023). Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition. In H. Bouamor, J. Pino, & K. Bali (Eds.), EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings (pp. 10007-10016). (EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings). Association for Computational Linguistics (ACL).

Radhakrishnan, Srijith ; Yang, Chao Han Huck ; Khan, Sumeer Ahmad et al. / Whispering LLaMA : A Cross-Modal Generative Error Correction Framework for Speech Recognition. EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings. editor / Houda Bouamor ; Juan Pino ; Kalika Bali. Association for Computational Linguistics (ACL), 2023. pp. 10007-10016 (EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings).

@inproceedings{258cf302e184496bae8ca88c9677fc40,

title = "Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition",

abstract = "We introduce a new cross-modal fusion technique designed for generative error correction in automatic speech recognition (ASR). Our methodology leverages both acoustic information and external linguistic representations to generate accurate speech transcription contexts. This marks a step towards a fresh paradigm in generative error correction within the realm of n-best hypotheses. Unlike the existing ranking-based rescoring methods, our approach adeptly uses distinct initialization techniques and parameter-efficient algorithms to boost ASR performance derived from pre-trained speech and text models. Through evaluation across diverse ASR datasets, we assess our fusion technique, demonstrating a 37.66% improvement in word error rate (WER) relative performance compared to the n-best Oracle. To encourage future research, we have made our code and pre-trained models open source at https://github.com/Srijith-rkr/Whispering-LLaMA.",

author = "Srijith Radhakrishnan and Yang, {Chao Han Huck} and Khan, {Sumeer Ahmad} and Rohit Kumar and Kiani, {Narsis A.} and David Gomez-Cabrero and Tegner, {Jesper N.}",

note = "Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.; 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 ; Conference date: 06-12-2023 Through 10-12-2023",

year = "2023",

language = "English (US)",

series = "EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings",

publisher = "Association for Computational Linguistics (ACL)",

pages = "10007--10016",

editor = "Houda Bouamor and Juan Pino and Kalika Bali",

booktitle = "EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings",

address = "United States",

}

Radhakrishnan, S, Yang, CHH, Khan, SA , Kumar, R, Kiani, NA, Gomez-Cabrero, D & Tegner, JN 2023, Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition. in H Bouamor, J Pino & K Bali (eds), EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings. EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings, Association for Computational Linguistics (ACL), pp. 10007-10016, 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Hybrid, Singapore, Singapore, 12/6/23.

Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition. / Radhakrishnan, Srijith; Yang, Chao Han Huck; Khan, Sumeer Ahmad et al.
EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings. ed. / Houda Bouamor; Juan Pino; Kalika Bali. Association for Computational Linguistics (ACL), 2023. p. 10007-10016 (EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Whispering LLaMA

T2 - 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023

AU - Radhakrishnan, Srijith

AU - Yang, Chao Han Huck

AU - Khan, Sumeer Ahmad

AU - Kumar, Rohit

AU - Kiani, Narsis A.

AU - Gomez-Cabrero, David

AU - Tegner, Jesper N.

PY - 2023

Y1 - 2023

N2 - We introduce a new cross-modal fusion technique designed for generative error correction in automatic speech recognition (ASR). Our methodology leverages both acoustic information and external linguistic representations to generate accurate speech transcription contexts. This marks a step towards a fresh paradigm in generative error correction within the realm of n-best hypotheses. Unlike the existing ranking-based rescoring methods, our approach adeptly uses distinct initialization techniques and parameter-efficient algorithms to boost ASR performance derived from pre-trained speech and text models. Through evaluation across diverse ASR datasets, we assess our fusion technique, demonstrating a 37.66% improvement in word error rate (WER) relative performance compared to the n-best Oracle. To encourage future research, we have made our code and pre-trained models open source at https://github.com/Srijith-rkr/Whispering-LLaMA.

AB - We introduce a new cross-modal fusion technique designed for generative error correction in automatic speech recognition (ASR). Our methodology leverages both acoustic information and external linguistic representations to generate accurate speech transcription contexts. This marks a step towards a fresh paradigm in generative error correction within the realm of n-best hypotheses. Unlike the existing ranking-based rescoring methods, our approach adeptly uses distinct initialization techniques and parameter-efficient algorithms to boost ASR performance derived from pre-trained speech and text models. Through evaluation across diverse ASR datasets, we assess our fusion technique, demonstrating a 37.66% improvement in word error rate (WER) relative performance compared to the n-best Oracle. To encourage future research, we have made our code and pre-trained models open source at https://github.com/Srijith-rkr/Whispering-LLaMA.

UR - http://www.scopus.com/inward/record.url?scp=85183818183&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85183818183

T3 - EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings

SP - 10007

EP - 10016

BT - EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings

A2 - Bouamor, Houda

A2 - Pino, Juan

A2 - Bali, Kalika

PB - Association for Computational Linguistics (ACL)

Y2 - 6 December 2023 through 10 December 2023

ER -

Radhakrishnan S, Yang CHH, Khan SA , Kumar R, Kiani NA, Gomez-Cabrero D et al. Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition. In Bouamor H, Pino J, Bali K, editors, EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings. Association for Computational Linguistics (ACL). 2023. p. 10007-10016. (EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings).

Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition

Abstract

Publication series

Conference

Bibliographical note

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this