Notified Access: Extending Remote Memory Access Programming Models for Producer-Consumer Synchronization

Roberto Belli, Torsten Hoefler

Research output: Chapter in Book/Report/Conference proceedingConference contribution

34 Scopus citations


Remote Memory Access (RMA) programming enables direct access to low-level hardware features to achieve high performance for distributed-memory programs. However, the design of RMA programming schemes focuses on the memory access and less on the synchronization. For example, in contemporary RMA programming systems, the widely used producer-consumer pattern can only be implemented inefficiently, incurring in an overhead of an additional round-trip message. We propose Notified Access, a scheme where the target process of an access can receive a completion notification. This scheme enables direct and efficient synchronization with a minimum number of messages. We implement our scheme in an open source MPI-3 RMA library and demonstrate lower overheads (two cache misses) than other point-to-point synchronization mechanisms for each notification. We also evaluate our implementation on three real-world benchmarks, a stencil computation, a tree computation, and a Colicky factorization implemented with tasks. Our scheme always performs better than traditional message passing and other existing RMA synchronization schemes, providing up to 50% speedup on small messages. Our analysis shows that Notified Access is a valuable primitive for any RMA system. Furthermore, we provide guidance for the design of low-level network interfaces to support Notified Access efficiently.
Original languageEnglish (US)
Title of host publication2015 IEEE International Parallel and Distributed Processing Symposium
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Number of pages11
ISBN (Print)9781479986491
StatePublished - May 2015
Externally publishedYes

Bibliographical note

KAUST Repository Item: Exported on 2020-10-01
Acknowledgements: We thank Hatem Ltaief (Kaust) for providing theCholesky example. We thank the GASPI team for inspir-ing discussions about RMA interfaces and Christian Sim-mendinger for numerous clarifications about the GASPIspecification. We thank James Dinan (Intel), Jeff Ham-mond (Intel), Kathy Yelick (LBNL), Edgar Solomonik, TimoSchneider, and Salvatore Di Girolamo for helpful discus-sions, Larry Kaplan (Cray) for help with uGNI, and theSwiss National Supercomputing Centre (CSCS) for accessto Piz Daint.
This publication acknowledges KAUST support, but has no KAUST affiliated authors.


Dive into the research topics of 'Notified Access: Extending Remote Memory Access Programming Models for Producer-Consumer Synchronization'. Together they form a unique fingerprint.

Cite this