Optimal Convergence Rates for the Orthogonal Greedy Algorithm

Jonathan W. Siegel, Jinchao Xu

Research output: Contribution to journalArticlepeer-review

7 Scopus citations

Abstract

We analyze the orthogonal greedy algorithm when applied to dictionaries D whose convex hull has small entropy. We show that if the metric entropy of the convex hull of D decays at a rate of O(n-frac 1 2-α ) for α > 0 , then the orthogonal greedy algorithm converges at the same rate on the variation space of D. This improves upon the well-known O (n-\frac 1 2) convergence rate of the orthogonal greedy algorithm in many cases, most notably for dictionaries corresponding to shallow neural networks. These results hold under no additional assumptions on the dictionary beyond the decay rate of the entropy of its convex hull. In addition, they are robust to noise in the target function and can be extended to convergence rates on the interpolation spaces of the variation norm. We show empirically that the predicted rates are obtained for the dictionary corresponding to shallow neural networks with Heaviside activation function in two dimensions. Finally, we show that these improved rates are sharp and prove a negative result showing that the iterates generated by the orthogonal greedy algorithm cannot in general be bounded in the variation norm of D.
Original languageEnglish (US)
Pages (from-to)3354-3361
Number of pages8
JournalIEEE Transactions on Information Theory
Volume68
Issue number5
DOIs
StatePublished - May 1 2022
Externally publishedYes

Bibliographical note

Generated from Scopus record by KAUST IRTS on 2023-02-15

Fingerprint

Dive into the research topics of 'Optimal Convergence Rates for the Orthogonal Greedy Algorithm'. Together they form a unique fingerprint.

Cite this