shmem4py: High-Performance One-Sided Communication for Python Applications

Marcin Rogowski; Jeff R. Hammond; David E. Keyes; Lisandro Dalcin

doi:10.1145/3624062.3624602

shmem4py: High-Performance One-Sided Communication for Python Applications

Marcin Rogowski, Jeff R. Hammond, David E. Keyes, Lisandro Dalcin

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

This paper describes shmem4py, a Python wrapper for the OpenSHMEM application programming interface (API) which follows a design similar to that of the well-known mpi4py package. OpenSHMEM is a descendant of the one-sided communication library for the Cray T3D and it is known for its uncompromising performance for low-latency and high-throughput use cases involving one-sided and collective communication. OpenSHMEM is arguably one of the most efficient and portable abstractions for modern network architectures. Thanks to tight interoperability with NumPy, shmem4py provides a convenient parallel programming framework leveraging both the high-productivity NumPy feature set and the high-performance networking capabilities of OpenSHMEM. This paper discusses the design and performance characteristics of shmem4py in a variety of communication patterns relative to lower-level languages (C) as well as MPI and mpi4py.

Original language	English (US)
Title of host publication	Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023
Publisher	Association for Computing Machinery
Pages	1185-1193
Number of pages	9
ISBN (Electronic)	9798400707858
DOIs	https://doi.org/10.1145/3624062.3624602
State	Published - Nov 12 2023
Event	2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023 - Denver, United States Duration: Nov 12 2023 → Nov 17 2023

Publication series

Name	ACM International Conference Proceeding Series

Conference

Conference	2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023
Country/Territory	United States
City	Denver
Period	11/12/23 → 11/17/23

Bibliographical note

Publisher Copyright:
© 2023 Owner/Author.

Keywords

High Performance Computing
MPI
OpenSHMEM
Python
shared memory

ASJC Scopus subject areas

Human-Computer Interaction
Computer Networks and Communications
Computer Vision and Pattern Recognition
Software

Access to Document

10.1145/3624062.3624602

Cite this

Rogowski, M., Hammond, J. R., Keyes, D. E., & Dalcin, L. (2023). shmem4py: High-Performance One-Sided Communication for Python Applications. In Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023 (pp. 1185-1193). (ACM International Conference Proceeding Series). Association for Computing Machinery. https://doi.org/10.1145/3624062.3624602

Rogowski, Marcin ; Hammond, Jeff R. ; Keyes, David E. et al. / shmem4py : High-Performance One-Sided Communication for Python Applications. Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023. Association for Computing Machinery, 2023. pp. 1185-1193 (ACM International Conference Proceeding Series).

@inproceedings{6257a7e50d40433da458badb4e54f5ba,

title = "shmem4py: High-Performance One-Sided Communication for Python Applications",

abstract = "This paper describes shmem4py, a Python wrapper for the OpenSHMEM application programming interface (API) which follows a design similar to that of the well-known mpi4py package. OpenSHMEM is a descendant of the one-sided communication library for the Cray T3D and it is known for its uncompromising performance for low-latency and high-throughput use cases involving one-sided and collective communication. OpenSHMEM is arguably one of the most efficient and portable abstractions for modern network architectures. Thanks to tight interoperability with NumPy, shmem4py provides a convenient parallel programming framework leveraging both the high-productivity NumPy feature set and the high-performance networking capabilities of OpenSHMEM. This paper discusses the design and performance characteristics of shmem4py in a variety of communication patterns relative to lower-level languages (C) as well as MPI and mpi4py.",

keywords = "High Performance Computing, MPI, OpenSHMEM, Python, shared memory",

author = "Marcin Rogowski and Hammond, {Jeff R.} and Keyes, {David E.} and Lisandro Dalcin",

note = "Publisher Copyright: {\textcopyright} 2023 Owner/Author.; 2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023 ; Conference date: 12-11-2023 Through 17-11-2023",

year = "2023",

month = nov,

day = "12",

doi = "10.1145/3624062.3624602",

language = "English (US)",

series = "ACM International Conference Proceeding Series",

publisher = "Association for Computing Machinery",

pages = "1185--1193",

booktitle = "Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023",

}

Rogowski, M, Hammond, JR, Keyes, DE & Dalcin, L 2023, shmem4py: High-Performance One-Sided Communication for Python Applications. in Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023. ACM International Conference Proceeding Series, Association for Computing Machinery, pp. 1185-1193, 2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023, Denver, United States, 11/12/23. https://doi.org/10.1145/3624062.3624602

shmem4py: High-Performance One-Sided Communication for Python Applications. / Rogowski, Marcin; Hammond, Jeff R.; Keyes, David E. et al.
Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023. Association for Computing Machinery, 2023. p. 1185-1193 (ACM International Conference Proceeding Series).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - shmem4py

T2 - 2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023

AU - Rogowski, Marcin

AU - Hammond, Jeff R.

AU - Keyes, David E.

AU - Dalcin, Lisandro

PY - 2023/11/12

Y1 - 2023/11/12

N2 - This paper describes shmem4py, a Python wrapper for the OpenSHMEM application programming interface (API) which follows a design similar to that of the well-known mpi4py package. OpenSHMEM is a descendant of the one-sided communication library for the Cray T3D and it is known for its uncompromising performance for low-latency and high-throughput use cases involving one-sided and collective communication. OpenSHMEM is arguably one of the most efficient and portable abstractions for modern network architectures. Thanks to tight interoperability with NumPy, shmem4py provides a convenient parallel programming framework leveraging both the high-productivity NumPy feature set and the high-performance networking capabilities of OpenSHMEM. This paper discusses the design and performance characteristics of shmem4py in a variety of communication patterns relative to lower-level languages (C) as well as MPI and mpi4py.

AB - This paper describes shmem4py, a Python wrapper for the OpenSHMEM application programming interface (API) which follows a design similar to that of the well-known mpi4py package. OpenSHMEM is a descendant of the one-sided communication library for the Cray T3D and it is known for its uncompromising performance for low-latency and high-throughput use cases involving one-sided and collective communication. OpenSHMEM is arguably one of the most efficient and portable abstractions for modern network architectures. Thanks to tight interoperability with NumPy, shmem4py provides a convenient parallel programming framework leveraging both the high-productivity NumPy feature set and the high-performance networking capabilities of OpenSHMEM. This paper discusses the design and performance characteristics of shmem4py in a variety of communication patterns relative to lower-level languages (C) as well as MPI and mpi4py.

KW - High Performance Computing

KW - MPI

KW - OpenSHMEM

KW - Python

KW - shared memory

UR - http://www.scopus.com/inward/record.url?scp=85178150246&partnerID=8YFLogxK

U2 - 10.1145/3624062.3624602

DO - 10.1145/3624062.3624602

M3 - Conference contribution

AN - SCOPUS:85178150246

T3 - ACM International Conference Proceeding Series

SP - 1185

EP - 1193

BT - Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023

PB - Association for Computing Machinery

Y2 - 12 November 2023 through 17 November 2023

ER -

Rogowski M, Hammond JR, Keyes DE , Dalcin L. shmem4py: High-Performance One-Sided Communication for Python Applications. In Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023. Association for Computing Machinery. 2023. p. 1185-1193. (ACM International Conference Proceeding Series). doi: 10.1145/3624062.3624602

shmem4py: High-Performance One-Sided Communication for Python Applications

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this