Relation-aware Video Reading Comprehension for Temporal Language Grounding

Jialin Gao; Xin Sun; Meng Meng Xu; Xi Zhou; Bernard Ghanem

Relation-aware Video Reading Comprehension for Temporal Language Grounding

Jialin Gao^*, Xin Sun^*, Meng Meng Xu, Xi Zhou, Bernard Ghanem

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

33 Scopus citations

Abstract

Temporal language grounding in videos aims to localize the temporal span relevant to the given query sentence. Previous methods treat it either as a boundary regression task or a span extraction task. This paper will formulate temporal language grounding into video reading comprehension and propose a Relation-aware Network (RaNet) to address it. This framework aims to select a video moment choice from the predefined answer set with the aid of coarse-and-fine choice-query interaction and choice-choice relation construction. A choice-query interactor is proposed to match the visual and textual information simultaneously in sentence-moment and token-moment levels, leading to a coarse-and-fine cross-modal interaction. Moreover, a novel multi-choice relation constructor is introduced by leveraging graph convolution to capture the dependencies among video moment choices for the best choice selection. Extensive experiments on ActivityNet-Captions, TACoS, and Charades-STA demonstrate the effectiveness of our solution. Codes will be available at https://github.com/Huntersxsx/RaNet.

Original language	English (US)
Title of host publication	EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings
Publisher	Association for Computational Linguistics (ACL)
Pages	3978-3988
Number of pages	11
ISBN (Electronic)	9781955917094
State	Published - 2021
Event	2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021 - Virtual, Punta Cana, Dominican Republic Duration: Nov 7 2021 → Nov 11 2021

Publication series

Name	EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings

Conference

Conference	2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021
Country/Territory	Dominican Republic
City	Virtual, Punta Cana
Period	11/7/21 → 11/11/21

Bibliographical note

Funding Information:
First of all, I would like to give my heartfelt thanks to all the people who have ever helped me in this paper. The support from CloudWalk Technology Co., Ltd is gratefully acknowledged. This work was also supported by the King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research through the Visual Computing Center (VCC) funding.

Publisher Copyright:
© 2021 Association for Computational Linguistics

ASJC Scopus subject areas

Computational Theory and Mathematics
Computer Science Applications
Information Systems

Cite this

Gao, J., Sun, X., Xu, M. M., Zhou, X., & Ghanem, B. (2021). Relation-aware Video Reading Comprehension for Temporal Language Grounding. In EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings (pp. 3978-3988). (EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings). Association for Computational Linguistics (ACL).

Gao, Jialin ; Sun, Xin ; Xu, Meng Meng et al. / Relation-aware Video Reading Comprehension for Temporal Language Grounding. EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings. Association for Computational Linguistics (ACL), 2021. pp. 3978-3988 (EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings).

@inproceedings{542a6927025442809e2402cd79ed83ca,

title = "Relation-aware Video Reading Comprehension for Temporal Language Grounding",

abstract = "Temporal language grounding in videos aims to localize the temporal span relevant to the given query sentence. Previous methods treat it either as a boundary regression task or a span extraction task. This paper will formulate temporal language grounding into video reading comprehension and propose a Relation-aware Network (RaNet) to address it. This framework aims to select a video moment choice from the predefined answer set with the aid of coarse-and-fine choice-query interaction and choice-choice relation construction. A choice-query interactor is proposed to match the visual and textual information simultaneously in sentence-moment and token-moment levels, leading to a coarse-and-fine cross-modal interaction. Moreover, a novel multi-choice relation constructor is introduced by leveraging graph convolution to capture the dependencies among video moment choices for the best choice selection. Extensive experiments on ActivityNet-Captions, TACoS, and Charades-STA demonstrate the effectiveness of our solution. Codes will be available at https://github.com/Huntersxsx/RaNet.",

author = "Jialin Gao and Xin Sun and Xu, {Meng Meng} and Xi Zhou and Bernard Ghanem",

note = "Funding Information: First of all, I would like to give my heartfelt thanks to all the people who have ever helped me in this paper. The support from CloudWalk Technology Co., Ltd is gratefully acknowledged. This work was also supported by the King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research through the Visual Computing Center (VCC) funding. Publisher Copyright: {\textcopyright} 2021 Association for Computational Linguistics; 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021 ; Conference date: 07-11-2021 Through 11-11-2021",

year = "2021",

language = "English (US)",

series = "EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings",

publisher = "Association for Computational Linguistics (ACL)",

pages = "3978--3988",

booktitle = "EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings",

address = "United States",

}

Gao, J, Sun, X, Xu, MM, Zhou, X & Ghanem, B 2021, Relation-aware Video Reading Comprehension for Temporal Language Grounding. in EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings. EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings, Association for Computational Linguistics (ACL), pp. 3978-3988, 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual, Punta Cana, Dominican Republic, 11/7/21.

Relation-aware Video Reading Comprehension for Temporal Language Grounding. / Gao, Jialin; Sun, Xin; Xu, Meng Meng et al.
EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings. Association for Computational Linguistics (ACL), 2021. p. 3978-3988 (EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Relation-aware Video Reading Comprehension for Temporal Language Grounding

AU - Gao, Jialin

AU - Sun, Xin

AU - Xu, Meng Meng

AU - Zhou, Xi

AU - Ghanem, Bernard

N1 - Funding Information: First of all, I would like to give my heartfelt thanks to all the people who have ever helped me in this paper. The support from CloudWalk Technology Co., Ltd is gratefully acknowledged. This work was also supported by the King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research through the Visual Computing Center (VCC) funding. Publisher Copyright: © 2021 Association for Computational Linguistics

PY - 2021

Y1 - 2021

N2 - Temporal language grounding in videos aims to localize the temporal span relevant to the given query sentence. Previous methods treat it either as a boundary regression task or a span extraction task. This paper will formulate temporal language grounding into video reading comprehension and propose a Relation-aware Network (RaNet) to address it. This framework aims to select a video moment choice from the predefined answer set with the aid of coarse-and-fine choice-query interaction and choice-choice relation construction. A choice-query interactor is proposed to match the visual and textual information simultaneously in sentence-moment and token-moment levels, leading to a coarse-and-fine cross-modal interaction. Moreover, a novel multi-choice relation constructor is introduced by leveraging graph convolution to capture the dependencies among video moment choices for the best choice selection. Extensive experiments on ActivityNet-Captions, TACoS, and Charades-STA demonstrate the effectiveness of our solution. Codes will be available at https://github.com/Huntersxsx/RaNet.

AB - Temporal language grounding in videos aims to localize the temporal span relevant to the given query sentence. Previous methods treat it either as a boundary regression task or a span extraction task. This paper will formulate temporal language grounding into video reading comprehension and propose a Relation-aware Network (RaNet) to address it. This framework aims to select a video moment choice from the predefined answer set with the aid of coarse-and-fine choice-query interaction and choice-choice relation construction. A choice-query interactor is proposed to match the visual and textual information simultaneously in sentence-moment and token-moment levels, leading to a coarse-and-fine cross-modal interaction. Moreover, a novel multi-choice relation constructor is introduced by leveraging graph convolution to capture the dependencies among video moment choices for the best choice selection. Extensive experiments on ActivityNet-Captions, TACoS, and Charades-STA demonstrate the effectiveness of our solution. Codes will be available at https://github.com/Huntersxsx/RaNet.

UR - http://www.scopus.com/inward/record.url?scp=85120602062&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85120602062

T3 - EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings

SP - 3978

EP - 3988

BT - EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings

PB - Association for Computational Linguistics (ACL)

T2 - 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021

Y2 - 7 November 2021 through 11 November 2021

ER -

Gao J, Sun X, Xu MM, Zhou X, Ghanem B. Relation-aware Video Reading Comprehension for Temporal Language Grounding. In EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings. Association for Computational Linguistics (ACL). 2021. p. 3978-3988. (EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings).

Relation-aware Video Reading Comprehension for Temporal Language Grounding

Abstract

Publication series

Conference

Bibliographical note

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this