Natural Compression for Distributed Deep Learning

Samuel Horváth*, Chen Yu Ho, Horváth L'udovít, Atal Narayan Sahu, Marco Canini, Peter Richtárik

*Corresponding author for this work

Research output: Contribution to conferencePaperpeer-review

15 Scopus citations

Abstract

Modern deep learning models are often trained in parallel over a collection of distributed machines to reduce training time. In such settings, communication of model updates among machines becomes a significant performance bottleneck, and various lossy update compression techniques have been proposed to alleviate this problem. In this work, we introduce a new, simple yet theoretically and practically effective compression technique: natural compression (Cnat). Our technique is applied individually to all entries of the to-be-compressed update vector. It works by randomized rounding to the nearest (negative or positive) power of two, which can be computed in a “natural” way by ignoring the mantissa. We show that compared to no compression, Cnat increases the second moment of the compressed vector by not more than the tiny factor 9/8, which means that the effect of Cnat on the convergence speed of popular training algorithms, such as distributed SGD, is negligible. However, the communications savings enabled by Cnat are substantial, leading to 3-4× improvement in overall theoretical running time. For applications requiring more aggressive compression, we generalize Cnat to natural dithering, which we prove is exponentially better than the common random dithering technique. Our compression operators can be used on their own or in combination with existing operators for a more aggressive combined effect while offering new state-of-the-art theoretical and practical performance.

Original languageEnglish (US)
Pages129-141
Number of pages13
StatePublished - 2022
Event3rd Annual Conference on Mathematical and Scientific Machine Learning, MSML 2022 - Beijing, China
Duration: Aug 15 2022Aug 17 2022

Conference

Conference3rd Annual Conference on Mathematical and Scientific Machine Learning, MSML 2022
Country/TerritoryChina
CityBeijing
Period08/15/2208/17/22

Bibliographical note

Publisher Copyright:
© 2022 S. Horváth, C.-Y. Ho, ̌ Horváth, A.N. Sahu, M. Canini & P. Richtárik.

Keywords

  • Distibuted Optimization
  • Gradient Compression
  • Non-convex Optimization
  • Stochastic Optimization

ASJC Scopus subject areas

  • Artificial Intelligence
  • Software
  • Control and Systems Engineering
  • Statistics and Probability

Fingerprint

Dive into the research topics of 'Natural Compression for Distributed Deep Learning'. Together they form a unique fingerprint.

Cite this