Recurrent nets that time and count

Felix A. Gers, Jurgen Schmidhuber

Research output: Chapter in Book/Report/Conference proceedingConference contribution

414 Scopus citations

Abstract

The size of the time intervals between events conveys information essential for numerous sequential tasks such as motor control and rhythm detection. While Hidden Markov Models tend to ignore this information, recurrent neural networks (RNNs) can in principle learn to make use of it. We focus on Long Short-Term Memory (LSTM) because it usually outperforms other RNNs. Surprisingly, LSTM augmented by `peephole connections' from its internal cells to its multiplicative gates can learn the fine distinction between sequences of spikes separated by either 50 or 49 discrete time steps, without the help of any short training exemplars. Without external resets or teacher forcing or loss of performance on tasks reported earlier, our LSTM variant also learns to generate very stable sequences of highly nonlinear, precisely timed spikes. This makes LSTM a promising approach for real-world tasks that require to time and count.
Original languageEnglish (US)
Title of host publicationProceedings of the International Joint Conference on Neural Networks
PublisherIEEEPiscataway, NJ, United States
Pages189-194
Number of pages6
StatePublished - Jan 1 2000
Externally publishedYes

Fingerprint

Dive into the research topics of 'Recurrent nets that time and count'. Together they form a unique fingerprint.

Cite this