Smart Gradient - An Adaptive Technique for Improving Gradient Estimation

Esmail Abdul Fattah, Janet Van Niekerk, Haavard Rue

Research output: Contribution to journalArticlepeer-review


Computing the gradient of a function provides fundamental information about its behavior. This information is essential for several applications and algorithms across various fields. One common application that require gradients are optimization techniques such as stochastic gradient descent, Newton's method and trust region methods. However, these methods usually requires a numerical computation of the gradient at every iteration of the method which is prone to numerical errors. We propose a simple limited-memory technique for improving the accuracy of a numerically computed gradient in this gradient-based optimization framework by exploiting (1) a coordinate transformation of the gradient and (2) the history of previously taken descent directions. The method is verified empirically by extensive experimentation on both test functions and on real data applications. The proposed method is implemented in the R package smartGrad and in C++.
Original languageEnglish (US)
JournalAccepted in Foundations of Data Science
StatePublished - 2021

Bibliographical note

KAUST Repository Item: Exported on 2021-12-16


Dive into the research topics of 'Smart Gradient - An Adaptive Technique for Improving Gradient Estimation'. Together they form a unique fingerprint.

Cite this