Doubly Adaptive Scaled Algorithm for Machine Learning Using Second-Order Information

Majid Jahani, Sergey Rusakov, Zheng Shi, Peter Richtarik, Michael W. Mahoney, Martin Takáč

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Scopus citations


We present a novel adaptive optimization algorithm for large-scale machine learning problems. Equipped with a low-cost estimate of local curvature and Lipschitz smoothness, our method dynamically adapts the search direction and step-size. The search direction contains gradient information preconditioned by a well-scaled diagonal preconditioning matrix that captures the local curvature information. Our methodology does not require the tedious task of learning rate tuning, as the learning rate is updated automatically without adding an extra hyperparameter. We provide convergence guarantees on a comprehensive collection of optimization problems, including convex, strongly convex, and nonconvex problems, in both deterministic and stochastic regimes. We also conduct an extensive empirical evaluation on standard machine learning problems, justifying our algorithm's versatility and demonstrating its strong performance compared to other start-of-the-art first-order and second-order methods.
Original languageEnglish (US)
Title of host publication10th International Conference on Learning Representations, ICLR 2022
PublisherInternational Conference on Learning Representations, ICLR
StatePublished - Jan 29 2022

Bibliographical note

KAUST Repository Item: Exported on 2023-03-28
Acknowledgements: MT was partially supported by the NSF, under award numbers CCF:1618717/CCF:1740796. PR was supported by the KAUST Baseline Research Funding Scheme. MM would like to acknowledge the US NSF and ONR via its BRC on RandNLA for providing partial support of this work. Our conclusions do not necessarily reflect the position or the policy of our sponsors, and no official endorsement should be inferred.


Dive into the research topics of 'Doubly Adaptive Scaled Algorithm for Machine Learning Using Second-Order Information'. Together they form a unique fingerprint.

Cite this