A Computation and Communication Efficient Method for Distributed Nonconvex Problems in the Partial Participation Setting

Alexander Tyurin; Peter Richtárik

A Computation and Communication Efficient Method for Distributed Nonconvex Problems in the Partial Participation Setting

Research output: Contribution to conference › Paper › peer-review

Abstract

We present a new method that includes three key components of distributed optimization and federated learning: variance reduction of stochastic gradients, partial participation, and compressed communication. We prove that the new method has optimal oracle complexity and state-of-the-art communication complexity in the partial participation setting. Regardless of the communication compression feature, our method successfully combines variance reduction and partial participation: we get the optimal oracle complexity, never need the participation of all nodes, and do not require the bounded gradients (dissimilarity) assumption.

Original language	English (US)
State	Published - 2023
Event	37th Conference on Neural Information Processing Systems, NeurIPS 2023 - New Orleans, United States Duration: Dec 10 2023 → Dec 16 2023

Conference

Conference	37th Conference on Neural Information Processing Systems, NeurIPS 2023
Country/Territory	United States
City	New Orleans
Period	12/10/23 → 12/16/23

Bibliographical note

Publisher Copyright:
© 2023 Neural information processing systems foundation. All rights reserved.

ASJC Scopus subject areas

Computer Networks and Communications
Information Systems
Signal Processing

Cite this

@conference{e57083b0caca44829d79605037872e70,

title = "A Computation and Communication Efficient Method for Distributed Nonconvex Problems in the Partial Participation Setting",

abstract = "We present a new method that includes three key components of distributed optimization and federated learning: variance reduction of stochastic gradients, partial participation, and compressed communication. We prove that the new method has optimal oracle complexity and state-of-the-art communication complexity in the partial participation setting. Regardless of the communication compression feature, our method successfully combines variance reduction and partial participation: we get the optimal oracle complexity, never need the participation of all nodes, and do not require the bounded gradients (dissimilarity) assumption.",

author = "Alexander Tyurin and Peter Richt{\'a}rik",

note = "Publisher Copyright: {\textcopyright} 2023 Neural information processing systems foundation. All rights reserved.; 37th Conference on Neural Information Processing Systems, NeurIPS 2023 ; Conference date: 10-12-2023 Through 16-12-2023",

year = "2023",

language = "English (US)",

}

TY - CONF

T1 - A Computation and Communication Efficient Method for Distributed Nonconvex Problems in the Partial Participation Setting

AU - Tyurin, Alexander

AU - Richtárik, Peter

PY - 2023

Y1 - 2023

N2 - We present a new method that includes three key components of distributed optimization and federated learning: variance reduction of stochastic gradients, partial participation, and compressed communication. We prove that the new method has optimal oracle complexity and state-of-the-art communication complexity in the partial participation setting. Regardless of the communication compression feature, our method successfully combines variance reduction and partial participation: we get the optimal oracle complexity, never need the participation of all nodes, and do not require the bounded gradients (dissimilarity) assumption.

AB - We present a new method that includes three key components of distributed optimization and federated learning: variance reduction of stochastic gradients, partial participation, and compressed communication. We prove that the new method has optimal oracle complexity and state-of-the-art communication complexity in the partial participation setting. Regardless of the communication compression feature, our method successfully combines variance reduction and partial participation: we get the optimal oracle complexity, never need the participation of all nodes, and do not require the bounded gradients (dissimilarity) assumption.

UR - http://www.scopus.com/inward/record.url?scp=85185578515&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:85185578515

T2 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023

Y2 - 10 December 2023 through 16 December 2023

ER -

A Computation and Communication Efficient Method for Distributed Nonconvex Problems in the Partial Participation Setting

Abstract

Conference

Bibliographical note

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this