A Comprehensive Empirical Study of Heterogeneity in Federated Learning

Ahmed M. Abdelmoniem; Chen-Yu Ho; Pantelis Papageorgiou; Marco Canini

doi:10.1109/jiot.2023.3250275

A Comprehensive Empirical Study of Heterogeneity in Federated Learning

Ahmed M. Abdelmoniem, Chen-Yu Ho, Pantelis Papageorgiou, Marco Canini

Research output: Contribution to journal › Article › peer-review

35 Scopus citations

Abstract

Federated learning (FL) is becoming a popular paradigm for collaborative learning over distributed, private datasets owned by non-trusting entities. FL has seen successful deployment in production environments, and it has been adopted in services such as virtual keyboards, auto-completion, item recommendation, and several IoT applications. However, FL comes with the challenge of performing training over largely heterogeneous datasets, devices, and networks that are out of the control of the centralized FL server. Motivated by this inherent challenge, we aim to empirically characterize the impact of device and behavioral heterogeneity on the trained model. We conduct an extensive empirical study spanning nearly 1.5K unique configurations on five popular FL benchmarks. Our analysis shows that these sources of heterogeneity have a major impact on both model quality and fairness, causing up to 4.6× and 2.2× degradation in the quality and fairness, respectively, thus shedding light on the importance of considering heterogeneity in FL system design.

Original language	English (US)
Pages (from-to)	1-1
Number of pages	1
Journal	IEEE Internet of Things Journal
DOIs	https://doi.org/10.1109/jiot.2023.3250275
State	Published - Mar 7 2023

Bibliographical note

KAUST Repository Item: Exported on 2023-03-10
Acknowledged KAUST grant number(s): ORA-CRG10-2021-4699
Acknowledgements: The work was conducted in part while Ahmed was with KAUST, KSA and while Pantelis was on an internship at KAUST, KSA We thank Muhammad Bilal for his help during the execution of the work. This publication is based upon work supported by King Abdullah University of Science and Technology (KAUST) under Award No. ORA-CRG10-2021-4699.

ASJC Scopus subject areas

Signal Processing
Information Systems
Information Systems and Management
Computer Science Applications
Hardware and Architecture
Computer Networks and Communications

Access to Document

10.1109/jiot.2023.3250275

https://repository.kaust.edu.sa/bitstream/10754/690220/1/A_Comprehensive_Empirical_Study_of_Heterogeneity_in_Federated_Learning.pdf

Cite this

@article{9e20bd2f07534826869bc027a8ffe112,

title = "A Comprehensive Empirical Study of Heterogeneity in Federated Learning",

abstract = "Federated learning (FL) is becoming a popular paradigm for collaborative learning over distributed, private datasets owned by non-trusting entities. FL has seen successful deployment in production environments, and it has been adopted in services such as virtual keyboards, auto-completion, item recommendation, and several IoT applications. However, FL comes with the challenge of performing training over largely heterogeneous datasets, devices, and networks that are out of the control of the centralized FL server. Motivated by this inherent challenge, we aim to empirically characterize the impact of device and behavioral heterogeneity on the trained model. We conduct an extensive empirical study spanning nearly 1.5K unique configurations on five popular FL benchmarks. Our analysis shows that these sources of heterogeneity have a major impact on both model quality and fairness, causing up to 4.6× and 2.2× degradation in the quality and fairness, respectively, thus shedding light on the importance of considering heterogeneity in FL system design.",

author = "Abdelmoniem, {Ahmed M.} and Chen-Yu Ho and Pantelis Papageorgiou and Marco Canini",

note = "KAUST Repository Item: Exported on 2023-03-10 Acknowledged KAUST grant number(s): ORA-CRG10-2021-4699 Acknowledgements: The work was conducted in part while Ahmed was with KAUST, KSA and while Pantelis was on an internship at KAUST, KSA We thank Muhammad Bilal for his help during the execution of the work. This publication is based upon work supported by King Abdullah University of Science and Technology (KAUST) under Award No. ORA-CRG10-2021-4699.",

year = "2023",

month = mar,

day = "7",

doi = "10.1109/jiot.2023.3250275",

language = "English (US)",

pages = "1--1",

journal = "IEEE Internet of Things Journal",

issn = "2327-4662",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - JOUR

T1 - A Comprehensive Empirical Study of Heterogeneity in Federated Learning

AU - Abdelmoniem, Ahmed M.

AU - Ho, Chen-Yu

AU - Papageorgiou, Pantelis

AU - Canini, Marco

N1 - KAUST Repository Item: Exported on 2023-03-10 Acknowledged KAUST grant number(s): ORA-CRG10-2021-4699 Acknowledgements: The work was conducted in part while Ahmed was with KAUST, KSA and while Pantelis was on an internship at KAUST, KSA We thank Muhammad Bilal for his help during the execution of the work. This publication is based upon work supported by King Abdullah University of Science and Technology (KAUST) under Award No. ORA-CRG10-2021-4699.

PY - 2023/3/7

Y1 - 2023/3/7

N2 - Federated learning (FL) is becoming a popular paradigm for collaborative learning over distributed, private datasets owned by non-trusting entities. FL has seen successful deployment in production environments, and it has been adopted in services such as virtual keyboards, auto-completion, item recommendation, and several IoT applications. However, FL comes with the challenge of performing training over largely heterogeneous datasets, devices, and networks that are out of the control of the centralized FL server. Motivated by this inherent challenge, we aim to empirically characterize the impact of device and behavioral heterogeneity on the trained model. We conduct an extensive empirical study spanning nearly 1.5K unique configurations on five popular FL benchmarks. Our analysis shows that these sources of heterogeneity have a major impact on both model quality and fairness, causing up to 4.6× and 2.2× degradation in the quality and fairness, respectively, thus shedding light on the importance of considering heterogeneity in FL system design.

AB - Federated learning (FL) is becoming a popular paradigm for collaborative learning over distributed, private datasets owned by non-trusting entities. FL has seen successful deployment in production environments, and it has been adopted in services such as virtual keyboards, auto-completion, item recommendation, and several IoT applications. However, FL comes with the challenge of performing training over largely heterogeneous datasets, devices, and networks that are out of the control of the centralized FL server. Motivated by this inherent challenge, we aim to empirically characterize the impact of device and behavioral heterogeneity on the trained model. We conduct an extensive empirical study spanning nearly 1.5K unique configurations on five popular FL benchmarks. Our analysis shows that these sources of heterogeneity have a major impact on both model quality and fairness, causing up to 4.6× and 2.2× degradation in the quality and fairness, respectively, thus shedding light on the importance of considering heterogeneity in FL system design.

UR - http://hdl.handle.net/10754/690220

UR - https://ieeexplore.ieee.org/document/10061708/

U2 - 10.1109/jiot.2023.3250275

DO - 10.1109/jiot.2023.3250275

M3 - Article

SN - 2327-4662

SP - 1

EP - 1

JO - IEEE Internet of Things Journal

JF - IEEE Internet of Things Journal

ER -

A Comprehensive Empirical Study of Heterogeneity in Federated Learning

Abstract

Bibliographical note

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this