Adaptive feature abstraction for translating video to language

Yunchen Pu; Zhe Gan; Lawrence Carin; Martin Renqiang Min

Adaptive feature abstraction for translating video to language

Yunchen Pu, Zhe Gan, Lawrence Carin, Martin Renqiang Min

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

A new model for video captioning is developed, using a deep three-dimensional Convolutional Neural Network (C3D) as an encoder for videos and a Recurrent Neural Network (RNN) as a decoder for captions. A novel attention mechanism with spatiotemporal alignment is employed to adaptively and sequentially focus on different layers of CNN features (levels of feature “abstraction”), as well as local spatiotemporal regions of the feature maps at each layer. The proposed approach is evaluated on the YouTube2Text benchmark. Experimental results demonstrate quantitatively the effectiveness of our proposed adaptive spatiotemporal feature abstraction for translating videos to sentences with rich semantic structures.

Original language	English (US)
Title of host publication	5th International Conference on Learning Representations, ICLR 2017 - Workshop Track Proceedings
Publisher	International Conference on Learning Representations, ICLR
State	Published - Jan 1 2019
Externally published	Yes

Bibliographical note

Generated from Scopus record by KAUST IRTS on 2021-02-09

Cite this

@inproceedings{b57cc292eea64872aca03186258536f5,

title = "Adaptive feature abstraction for translating video to language",

abstract = "A new model for video captioning is developed, using a deep three-dimensional Convolutional Neural Network (C3D) as an encoder for videos and a Recurrent Neural Network (RNN) as a decoder for captions. A novel attention mechanism with spatiotemporal alignment is employed to adaptively and sequentially focus on different layers of CNN features (levels of feature “abstraction”), as well as local spatiotemporal regions of the feature maps at each layer. The proposed approach is evaluated on the YouTube2Text benchmark. Experimental results demonstrate quantitatively the effectiveness of our proposed adaptive spatiotemporal feature abstraction for translating videos to sentences with rich semantic structures.",

author = "Yunchen Pu and Zhe Gan and Lawrence Carin and Min, {Martin Renqiang}",

note = "Generated from Scopus record by KAUST IRTS on 2021-02-09",

year = "2019",

month = jan,

day = "1",

language = "English (US)",

booktitle = "5th International Conference on Learning Representations, ICLR 2017 - Workshop Track Proceedings",

publisher = "International Conference on Learning Representations, ICLR",

}

TY - GEN

T1 - Adaptive feature abstraction for translating video to language

AU - Pu, Yunchen

AU - Gan, Zhe

AU - Carin, Lawrence

AU - Min, Martin Renqiang

N1 - Generated from Scopus record by KAUST IRTS on 2021-02-09

PY - 2019/1/1

Y1 - 2019/1/1

N2 - A new model for video captioning is developed, using a deep three-dimensional Convolutional Neural Network (C3D) as an encoder for videos and a Recurrent Neural Network (RNN) as a decoder for captions. A novel attention mechanism with spatiotemporal alignment is employed to adaptively and sequentially focus on different layers of CNN features (levels of feature “abstraction”), as well as local spatiotemporal regions of the feature maps at each layer. The proposed approach is evaluated on the YouTube2Text benchmark. Experimental results demonstrate quantitatively the effectiveness of our proposed adaptive spatiotemporal feature abstraction for translating videos to sentences with rich semantic structures.

AB - A new model for video captioning is developed, using a deep three-dimensional Convolutional Neural Network (C3D) as an encoder for videos and a Recurrent Neural Network (RNN) as a decoder for captions. A novel attention mechanism with spatiotemporal alignment is employed to adaptively and sequentially focus on different layers of CNN features (levels of feature “abstraction”), as well as local spatiotemporal regions of the feature maps at each layer. The proposed approach is evaluated on the YouTube2Text benchmark. Experimental results demonstrate quantitatively the effectiveness of our proposed adaptive spatiotemporal feature abstraction for translating videos to sentences with rich semantic structures.

UR - http://www.scopus.com/inward/record.url?scp=85083953548&partnerID=8YFLogxK

M3 - Conference contribution

BT - 5th International Conference on Learning Representations, ICLR 2017 - Workshop Track Proceedings

PB - International Conference on Learning Representations, ICLR

ER -

Adaptive feature abstraction for translating video to language

Abstract

Bibliographical note

Other files and links

Fingerprint

Cite this