Deep networks with internal selective attention through feedback connections

Marijn F. Stollenga; Jonathan Masci; Faustino Gomez; Juergen Schmidhuber

Deep networks with internal selective attention through feedback connections

Marijn F. Stollenga, Jonathan Masci, Faustino Gomez, Juergen Schmidhuber

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Traditional convolutional neural networks (CNN) are stationary and feedforward. They neither change their parameters during evaluation nor use feedback from higher to lower layers. Real brains, however, do. So does our Deep Attention Selective Network (dasNet) architecture. DasNet's feedback structure can dynamically alter its convolutional filter sensitivities during classification. It harnesses the power of sequential processing to improve classification performance, by allowing the network to iteratively focus its internal attention on some of its convolutional filters. Feedback is trained through direct policy search in a huge million-dimensional parameter space, through scalable natural evolution strategies (SNES). On the CIFAR-10 and CIFAR-100 datasets, dasNet outperforms the previous state-of-the-art model on unaugmented datasets.

Original language	English (US)
Title of host publication	Advances in Neural Information Processing Systems
Publisher	Neural information processing systems foundation
Pages	3545-3553
Number of pages	9
State	Published - Jan 1 2014
Externally published	Yes

Bibliographical note

Generated from Scopus record by KAUST IRTS on 2022-09-14

Cite this

@inproceedings{087f608a7a2146e0bd45947f8168dfe9,

title = "Deep networks with internal selective attention through feedback connections",

abstract = "Traditional convolutional neural networks (CNN) are stationary and feedforward. They neither change their parameters during evaluation nor use feedback from higher to lower layers. Real brains, however, do. So does our Deep Attention Selective Network (dasNet) architecture. DasNet's feedback structure can dynamically alter its convolutional filter sensitivities during classification. It harnesses the power of sequential processing to improve classification performance, by allowing the network to iteratively focus its internal attention on some of its convolutional filters. Feedback is trained through direct policy search in a huge million-dimensional parameter space, through scalable natural evolution strategies (SNES). On the CIFAR-10 and CIFAR-100 datasets, dasNet outperforms the previous state-of-the-art model on unaugmented datasets.",

author = "Stollenga, {Marijn F.} and Jonathan Masci and Faustino Gomez and Juergen Schmidhuber",

note = "Generated from Scopus record by KAUST IRTS on 2022-09-14",

year = "2014",

month = jan,

day = "1",

language = "English (US)",

pages = "3545--3553",

booktitle = "Advances in Neural Information Processing Systems",

publisher = "Neural information processing systems foundation",

}

TY - GEN

T1 - Deep networks with internal selective attention through feedback connections

AU - Stollenga, Marijn F.

AU - Masci, Jonathan

AU - Gomez, Faustino

AU - Schmidhuber, Juergen

N1 - Generated from Scopus record by KAUST IRTS on 2022-09-14

PY - 2014/1/1

Y1 - 2014/1/1

N2 - Traditional convolutional neural networks (CNN) are stationary and feedforward. They neither change their parameters during evaluation nor use feedback from higher to lower layers. Real brains, however, do. So does our Deep Attention Selective Network (dasNet) architecture. DasNet's feedback structure can dynamically alter its convolutional filter sensitivities during classification. It harnesses the power of sequential processing to improve classification performance, by allowing the network to iteratively focus its internal attention on some of its convolutional filters. Feedback is trained through direct policy search in a huge million-dimensional parameter space, through scalable natural evolution strategies (SNES). On the CIFAR-10 and CIFAR-100 datasets, dasNet outperforms the previous state-of-the-art model on unaugmented datasets.

AB - Traditional convolutional neural networks (CNN) are stationary and feedforward. They neither change their parameters during evaluation nor use feedback from higher to lower layers. Real brains, however, do. So does our Deep Attention Selective Network (dasNet) architecture. DasNet's feedback structure can dynamically alter its convolutional filter sensitivities during classification. It harnesses the power of sequential processing to improve classification performance, by allowing the network to iteratively focus its internal attention on some of its convolutional filters. Feedback is trained through direct policy search in a huge million-dimensional parameter space, through scalable natural evolution strategies (SNES). On the CIFAR-10 and CIFAR-100 datasets, dasNet outperforms the previous state-of-the-art model on unaugmented datasets.

UR - http://www.scopus.com/inward/record.url?scp=84937961845&partnerID=8YFLogxK

M3 - Conference contribution

SP - 3545

EP - 3553

BT - Advances in Neural Information Processing Systems

PB - Neural information processing systems foundation

ER -

Deep networks with internal selective attention through feedback connections

Abstract

Bibliographical note

Other files and links

Fingerprint

Cite this