a Daptive F Eature a Bstraction

Zhe Gan, Lawrence Carin, Martin Renqiang Min

Research output: Contribution to journalArticlepeer-review


A new model for video captioning is developed, using a deep three-dimensional Convolutional Neural Network (C3D) as an encoder for videos and a Recurrent Neural Network (RNN) as a decoder for captions. A novel attention mechanism with spatiotemporal alignment is employed to adaptively and sequentially focus on different layers of CNN features (levels of feature " abstraction "), as well as local spatiotemporal regions of the feature maps at each layer. The proposed approach is evaluated on the YouTube2Text benchmark. Experimental results demonstrate quantitatively the effectiveness of our proposed adaptive spatiotem-poral feature abstraction for translating videos to sentences with rich semantic structures.
Original languageEnglish (US)
Pages (from-to)1-4
Number of pages4
Issue number2014
StatePublished - 2017
Externally publishedYes


Dive into the research topics of 'a Daptive F Eature a Bstraction'. Together they form a unique fingerprint.

Cite this