Neighborhood communication paradigm to increase scalability in large-scale dynamic scientific applications

Aleksandr Ovcharenko, Daniel Ibanez, Fabien Delalondre, Onkar Sahni, Kenneth E. Jansen, Christopher D. Carothers, Mark S. Shephard

Research output: Contribution to journalArticlepeer-review

17 Scopus citations


This paper introduces a general-purpose communication package built on top of MPI which is aimed at improving inter-processor communications independently of the supercomputer architecture being considered. The package is developed to support parallel applications that rely on computation characterized by large number of messages of various sizes, often small, that are focused within processor neighborhoods. In some cases, such as solvers having static mesh partitions, the number and size of messages are known a priori. However, in other cases such as mesh adaptation, the messages evolve and vary in number and size and include the dynamic movement of partition objects. The current package provides a utility for dynamic applications based on two key attributes that are: (i) explicit consideration of the neighborhood communication pattern to avoid many-to-many calls and also to reduce the number of collective calls to a minimum, and (ii) use of non-blocking MPI functions along with message packing to manage message flow control and reduce the number and time of communication calls. The test application demonstrated is parallel unstructured mesh adaptation. Results on IBM Blue Gene/P and Cray XE6 computers show that the use of neighborhood-based communication control leads to scalable results when executing generally imbalanced mesh adaptation runs. © 2011 Elsevier B.V. All rights reserved.
Original languageEnglish (US)
Pages (from-to)140-156
Number of pages17
JournalParallel Computing
Issue number3
StatePublished - Mar 2012
Externally publishedYes

Bibliographical note

KAUST Repository Item: Exported on 2020-10-01
Acknowledgements: This work is supported by the National Science Foundation under Grant No. 0749152, and by the US Department of Energy under DOE Grant No. DE-FC02-06ER25769. Computing support is provided by King Abdullah University of Science and Technology for granting access to the Blue Gene/P and by National Energy Research Scientific Computing Center for granting access to the Cray XE6 supercomputers.The authors thank Professor William D. Gropp for the valuable comments and clarifications towards the improvement of this study.
This publication acknowledges KAUST support, but has no KAUST affiliated authors.


Dive into the research topics of 'Neighborhood communication paradigm to increase scalability in large-scale dynamic scientific applications'. Together they form a unique fingerprint.

Cite this