Kimad: Adaptive Gradient Compression with Bandwidth Awareness

Jihao Xin; Ivan Ilin; Shunkang Zhang; Marco Canini; Peter Richtárik

doi:10.1145/3630048.3630184

Kimad: Adaptive Gradient Compression with Bandwidth Awareness

Jihao Xin, Ivan Ilin, Shunkang Zhang, Marco Canini, Peter Richtárik

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

2 Scopus citations

Abstract

In distributed training, communication often emerges as a bottleneck. In response, we introduce Kimad, a solution that offers adaptive gradient compression. By consistently monitoring bandwidth, Kimad refines compression ratios to match specific neural network layer requirements. Our exhaustive tests and proofs confirm Kimad's outstanding performance, establishing it as a benchmark in adaptive compression for distributed deep learning.

Original language	English (US)
Title of host publication	DistributedML 2023 - Proceedings of the 4th International Workshop on Distributed Machine Learning
Publisher	Association for Computing Machinery, Inc
Pages	35-48
Number of pages	14
ISBN (Electronic)	9798400704475
DOIs	https://doi.org/10.1145/3630048.3630184
State	Published - Dec 8 2023
Event	4th International Workshop on Distributed Machine Learning, DistributedML 2023 - Paris, France Duration: Dec 8 2023 → …

Publication series

Name	DistributedML 2023 - Proceedings of the 4th International Workshop on Distributed Machine Learning

Conference

Conference	4th International Workshop on Distributed Machine Learning, DistributedML 2023
Country/Territory	France
City	Paris
Period	12/8/23 → …

Bibliographical note

Publisher Copyright:
© 2023 Owner/Author.

Keywords

distributed training
gradient compression

ASJC Scopus subject areas

Computer Networks and Communications
Computer Science Applications
Hardware and Architecture

Access to Document

10.1145/3630048.3630184

Cite this

Xin, J., Ilin, I., Zhang, S., Canini, M., & Richtárik, P. (2023). Kimad: Adaptive Gradient Compression with Bandwidth Awareness. In DistributedML 2023 - Proceedings of the 4th International Workshop on Distributed Machine Learning (pp. 35-48). (DistributedML 2023 - Proceedings of the 4th International Workshop on Distributed Machine Learning). Association for Computing Machinery, Inc. https://doi.org/10.1145/3630048.3630184

Xin, Jihao ; Ilin, Ivan ; Zhang, Shunkang et al. / Kimad : Adaptive Gradient Compression with Bandwidth Awareness. DistributedML 2023 - Proceedings of the 4th International Workshop on Distributed Machine Learning. Association for Computing Machinery, Inc, 2023. pp. 35-48 (DistributedML 2023 - Proceedings of the 4th International Workshop on Distributed Machine Learning).

@inproceedings{34874a9c3d87481382715cc4552f2709,

title = "Kimad: Adaptive Gradient Compression with Bandwidth Awareness",

abstract = "In distributed training, communication often emerges as a bottleneck. In response, we introduce Kimad, a solution that offers adaptive gradient compression. By consistently monitoring bandwidth, Kimad refines compression ratios to match specific neural network layer requirements. Our exhaustive tests and proofs confirm Kimad's outstanding performance, establishing it as a benchmark in adaptive compression for distributed deep learning.",

keywords = "distributed training, gradient compression",

author = "Jihao Xin and Ivan Ilin and Shunkang Zhang and Marco Canini and Peter Richt{\'a}rik",

note = "Publisher Copyright: {\textcopyright} 2023 Owner/Author.; 4th International Workshop on Distributed Machine Learning, DistributedML 2023 ; Conference date: 08-12-2023",

year = "2023",

month = dec,

day = "8",

doi = "10.1145/3630048.3630184",

language = "English (US)",

series = "DistributedML 2023 - Proceedings of the 4th International Workshop on Distributed Machine Learning",

publisher = "Association for Computing Machinery, Inc",

pages = "35--48",

booktitle = "DistributedML 2023 - Proceedings of the 4th International Workshop on Distributed Machine Learning",

}

Xin, J , Ilin, I, Zhang, S, Canini, M & Richtárik, P 2023, Kimad: Adaptive Gradient Compression with Bandwidth Awareness. in DistributedML 2023 - Proceedings of the 4th International Workshop on Distributed Machine Learning. DistributedML 2023 - Proceedings of the 4th International Workshop on Distributed Machine Learning, Association for Computing Machinery, Inc, pp. 35-48, 4th International Workshop on Distributed Machine Learning, DistributedML 2023, Paris, France, 12/8/23. https://doi.org/10.1145/3630048.3630184

Kimad: Adaptive Gradient Compression with Bandwidth Awareness. / Xin, Jihao ; Ilin, Ivan; Zhang, Shunkang et al.
DistributedML 2023 - Proceedings of the 4th International Workshop on Distributed Machine Learning. Association for Computing Machinery, Inc, 2023. p. 35-48 (DistributedML 2023 - Proceedings of the 4th International Workshop on Distributed Machine Learning).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Kimad

T2 - 4th International Workshop on Distributed Machine Learning, DistributedML 2023

AU - Xin, Jihao

AU - Ilin, Ivan

AU - Zhang, Shunkang

AU - Canini, Marco

AU - Richtárik, Peter

PY - 2023/12/8

Y1 - 2023/12/8

N2 - In distributed training, communication often emerges as a bottleneck. In response, we introduce Kimad, a solution that offers adaptive gradient compression. By consistently monitoring bandwidth, Kimad refines compression ratios to match specific neural network layer requirements. Our exhaustive tests and proofs confirm Kimad's outstanding performance, establishing it as a benchmark in adaptive compression for distributed deep learning.

AB - In distributed training, communication often emerges as a bottleneck. In response, we introduce Kimad, a solution that offers adaptive gradient compression. By consistently monitoring bandwidth, Kimad refines compression ratios to match specific neural network layer requirements. Our exhaustive tests and proofs confirm Kimad's outstanding performance, establishing it as a benchmark in adaptive compression for distributed deep learning.

KW - distributed training

KW - gradient compression

UR - http://www.scopus.com/inward/record.url?scp=85183575903&partnerID=8YFLogxK

U2 - 10.1145/3630048.3630184

DO - 10.1145/3630048.3630184

M3 - Conference contribution

AN - SCOPUS:85183575903

T3 - DistributedML 2023 - Proceedings of the 4th International Workshop on Distributed Machine Learning

SP - 35

EP - 48

BT - DistributedML 2023 - Proceedings of the 4th International Workshop on Distributed Machine Learning

PB - Association for Computing Machinery, Inc

Y2 - 8 December 2023

ER -

Xin J , Ilin I, Zhang S, Canini M , Richtárik P. Kimad: Adaptive Gradient Compression with Bandwidth Awareness. In DistributedML 2023 - Proceedings of the 4th International Workshop on Distributed Machine Learning. Association for Computing Machinery, Inc. 2023. p. 35-48. (DistributedML 2023 - Proceedings of the 4th International Workshop on Distributed Machine Learning). doi: 10.1145/3630048.3630184

Kimad: Adaptive Gradient Compression with Bandwidth Awareness

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this