Flat minima

Sepp Hochreiter, Jürgen Schmidhuber

Research output: Contribution to journalArticlepeer-review

313 Scopus citations

Abstract

We present a new algorithm for finding low-complexity neural networks with high generalization capability. The algorithm searches for a "flat" minimum of the error function. A flat minimum is a large connected region in weight space where the error remains approximately constant. An MDL-based, Bayesian argument suggests that flat minima correspond to "simple" networks and low expected overfitting. The argument is based on a Gibbs algorithm variant and a novel way of splitting generalization error into underfilling and overfitting error. Unlike many previous approaches, ours does not require gaussian assumptions and does not depend on a "good" weight prior. Instead we have a prior over input-output functions, thus taking into account net architecture and training set. Although our algorithm requires the computation of second-order derivatives, it has backpropagation's order of complexity. Automatically, it effectively prunes units, weights, and input lines. Various experiments with feedforward and recurrent nets are described. In an application to stock market prediction, flat minimum search outperforms conventional backprop, weight decay, and "optimal brain surgeon/optimal brain damage.".
Original languageEnglish (US)
Pages (from-to)1-42
Number of pages42
JournalNeural Computation
Volume9
Issue number1
DOIs
StatePublished - Jan 1 1997
Externally publishedYes

ASJC Scopus subject areas

  • Cognitive Neuroscience

Fingerprint

Dive into the research topics of 'Flat minima'. Together they form a unique fingerprint.

Cite this