A Computable Definition of the Spectral Bias

Jonas Kiessling, Filip Thor

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations


Neural networks have a bias towards low frequency functions. This spectral bias has been the subject of several previous studies, both empirical and theoretical. Here we present a computable definition of the spectral bias based on a decomposition of the reconstruction error into a low and a high frequency component. The distinction between low and high frequencies is made in a way that allows for easy interpretation of the spectral bias. Furthermore, we present two methods for estimating the spectral bias. Method 1 relies on the use of the discrete Fourier transform to explicitly estimate the Fourier spectrum of the prediction residual, and Method 2 uses convolution to extract the low frequency components, where the convolution integral is estimated by Monte Carlo methods. The spectral bias depends on the distribution of the data, which is approximated with kernel density estimation when unknown. We devise a set of numerical experiments that confirm that low frequencies are learned first, a behavior quantified by our definition.
Original languageEnglish (US)
Title of host publicationProceedings of the AAAI Conference on Artificial Intelligence
PublisherAssociation for the Advancement of Artificial Intelligence (AAAI)
Number of pages8
ISBN (Print)1577358767
StatePublished - Jun 28 2022
Externally publishedYes

Bibliographical note

KAUST Repository Item: Exported on 2023-03-02
Acknowledged KAUST grant number(s): OSR-2019-CRG8-4033.2
Acknowledgements: This work was partially supported by the KAUST Office of Sponsored Research (OSR) under Award numbers OSR-2019-CRG8-4033.2.
This publication acknowledges KAUST support, but has no KAUST affiliated authors.


Dive into the research topics of 'A Computable Definition of the Spectral Bias'. Together they form a unique fingerprint.

Cite this