Searching for minimal neural networks in fourier space

Jan Koutník, Faustino Gomez, Jürgen Schmidhuber

Research output: Chapter in Book/Report/Conference proceedingConference contribution

9 Scopus citations

Abstract

The principle of minimum description length suggests looking for the simplest network that works well on the training examples, where simplicity is measured by network description size based on a reasonable programming language for encoding networks. Previous work used an assembler-like universal network encoding language (NEL) and Speed Prior-based search (related to Levin's Universal Search) to quickly find low-complexity nets with excellent generalization performance. Here we define a more natural and often more practical NEL whose instructions are frequency domain coefficients. Frequency coefficients may get encoded by few bits, hence huge weight matrices may just be low-complexity superpositions of patterns computed by programs with few elementary instructions. On various benchmarks this weight matrix encoding greatly accelerates the search. The scheme was tested on pole-balancing, long-term dependency T-maze, and ball throwing. Some of the solutions turn out to be unexpectedly simple as they are computable by fairly short bit strings.
Original languageEnglish (US)
Title of host publicationArtificial General Intelligence - Proceedings of the Third Conference on Artificial General Intelligence, AGI 2010
PublisherAtlantis Press
Pages61-66
Number of pages6
ISBN (Print)9789078677369
DOIs
StatePublished - Jan 1 2010
Externally publishedYes

Bibliographical note

Generated from Scopus record by KAUST IRTS on 2022-09-14

Fingerprint

Dive into the research topics of 'Searching for minimal neural networks in fourier space'. Together they form a unique fingerprint.

Cite this