Assise: Performance and Availability via Client-local NVM in a Distributed File System

Thomas E. Anderson; Marco Canini; Jongyul Kim; Dejan Kostic; Youngjin Kwon; Simon Peter; Waleed Reda; Henry N. Schuh; Emmett Witchel

Assise: Performance and Availability via Client-local NVM in a Distributed File System

Thomas E. Anderson, Marco Canini, Jongyul Kim, Dejan Kostic, Youngjin Kwon, Simon Peter, Waleed Reda, Henry N. Schuh, Emmett Witchel

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

he adoption of low latency persistent memory modules (PMMs) upends the long-established model of remote storage for distributed file systems. Instead, by colocating computation with PMM storage, we can provide applications with much higher IO performance, sub-second application failover, and strong consistency. To demonstrate this, we built the Assise distributed file system, based on a persistent, replicated coherence protocol that manages client-local PMM as a linearizable and crash-recoverable cache between applications and slower (and possibly remote) storage. Assise maximizes locality for all file IO by carrying out IO on process-local, socket-local, and client-local PMM whenever possible. Assise minimizes coherence overhead by maintaining consistency at IO operation granularity, rather than at fixed block sizes. We compare Assise to Ceph/BlueStore, NFS, and Octopus on a cluster with Intel Optane DC PMMs and SSDs for common cloud applications and benchmarks, such as LevelDB, Postfix, and FileBench. We find that Assise improves write latency up to 22x, throughput up to 56x, fail-over time up to 103x, and scales up to 6x better than its counterparts, while providing stronger consistency semantics.

Original language	English (US)
Title of host publication	Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation
Pages	1011-1027
Number of pages	17
State	Published - 2020

Bibliographical note

KAUST Repository Item: Exported on 2021-08-12
Acknowledgements: Waleed Reda was supported by a fellowship from the Erasmus Mundus Joint Doctorate in Distributed Computing (EMJD-DC), funded by the European Commission (EACEA) (FPA 2012-0030). This work is supported in part by ERC grant 770889, NSF grant CNS-1900457, and the Texas Systems Research Consortium. This work is also supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2020R1C1C1014940). We thank Intel for access to the evaluation testbed. We thank the anonymous reviewers and our shepherd, Kim Keeton, for their comments and feedback.

Access to Document

https://repository.kaust.edu.sa/bitstream/10754/670570/1/Conference%20Paperfile1.pdf

Cite this

Anderson, T. E., Canini, M., Kim, J., Kostic, D., Kwon, Y., Peter, S., Reda, W., Schuh, H. N., & Witchel, E. (2020). Assise: Performance and Availability via Client-local NVM in a Distributed File System. In Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (pp. 1011-1027) https://repository.kaust.edu.sa/bitstream/10754/670570/1/Conference%20Paperfile1.pdf

@inproceedings{77d142c18e0e47d695edcd0bcdadf465,

title = "Assise: Performance and Availability via Client-local NVM in a Distributed File System",

abstract = "he adoption of low latency persistent memory modules (PMMs) upends the long-established model of remote storage for distributed file systems. Instead, by colocating computation with PMM storage, we can provide applications with much higher IO performance, sub-second application failover, and strong consistency. To demonstrate this, we built the Assise distributed file system, based on a persistent, replicated coherence protocol that manages client-local PMM as a linearizable and crash-recoverable cache between applications and slower (and possibly remote) storage. Assise maximizes locality for all file IO by carrying out IO on process-local, socket-local, and client-local PMM whenever possible. Assise minimizes coherence overhead by maintaining consistency at IO operation granularity, rather than at fixed block sizes. We compare Assise to Ceph/BlueStore, NFS, and Octopus on a cluster with Intel Optane DC PMMs and SSDs for common cloud applications and benchmarks, such as LevelDB, Postfix, and FileBench. We find that Assise improves write latency up to 22x, throughput up to 56x, fail-over time up to 103x, and scales up to 6x better than its counterparts, while providing stronger consistency semantics.",

author = "Anderson, {Thomas E.} and Marco Canini and Jongyul Kim and Dejan Kostic and Youngjin Kwon and Simon Peter and Waleed Reda and Schuh, {Henry N.} and Emmett Witchel",

note = "KAUST Repository Item: Exported on 2021-08-12 Acknowledgements: Waleed Reda was supported by a fellowship from the Erasmus Mundus Joint Doctorate in Distributed Computing (EMJD-DC), funded by the European Commission (EACEA) (FPA 2012-0030). This work is supported in part by ERC grant 770889, NSF grant CNS-1900457, and the Texas Systems Research Consortium. This work is also supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2020R1C1C1014940). We thank Intel for access to the evaluation testbed. We thank the anonymous reviewers and our shepherd, Kim Keeton, for their comments and feedback.",

year = "2020",

language = "English (US)",

pages = "1011--1027",

booktitle = "Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation",

}

Anderson, TE, Canini, M, Kim, J, Kostic, D, Kwon, Y, Peter, S, Reda, W, Schuh, HN & Witchel, E 2020, Assise: Performance and Availability via Client-local NVM in a Distributed File System. in Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation. pp. 1011-1027. <https://repository.kaust.edu.sa/bitstream/10754/670570/1/Conference%20Paperfile1.pdf>

TY - GEN

T1 - Assise: Performance and Availability via Client-local NVM in a Distributed File System

AU - Anderson, Thomas E.

AU - Canini, Marco

AU - Kim, Jongyul

AU - Kostic, Dejan

AU - Kwon, Youngjin

AU - Peter, Simon

AU - Reda, Waleed

AU - Schuh, Henry N.

AU - Witchel, Emmett

N1 - KAUST Repository Item: Exported on 2021-08-12 Acknowledgements: Waleed Reda was supported by a fellowship from the Erasmus Mundus Joint Doctorate in Distributed Computing (EMJD-DC), funded by the European Commission (EACEA) (FPA 2012-0030). This work is supported in part by ERC grant 770889, NSF grant CNS-1900457, and the Texas Systems Research Consortium. This work is also supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2020R1C1C1014940). We thank Intel for access to the evaluation testbed. We thank the anonymous reviewers and our shepherd, Kim Keeton, for their comments and feedback.

PY - 2020

Y1 - 2020

N2 - he adoption of low latency persistent memory modules (PMMs) upends the long-established model of remote storage for distributed file systems. Instead, by colocating computation with PMM storage, we can provide applications with much higher IO performance, sub-second application failover, and strong consistency. To demonstrate this, we built the Assise distributed file system, based on a persistent, replicated coherence protocol that manages client-local PMM as a linearizable and crash-recoverable cache between applications and slower (and possibly remote) storage. Assise maximizes locality for all file IO by carrying out IO on process-local, socket-local, and client-local PMM whenever possible. Assise minimizes coherence overhead by maintaining consistency at IO operation granularity, rather than at fixed block sizes. We compare Assise to Ceph/BlueStore, NFS, and Octopus on a cluster with Intel Optane DC PMMs and SSDs for common cloud applications and benchmarks, such as LevelDB, Postfix, and FileBench. We find that Assise improves write latency up to 22x, throughput up to 56x, fail-over time up to 103x, and scales up to 6x better than its counterparts, while providing stronger consistency semantics.

AB - he adoption of low latency persistent memory modules (PMMs) upends the long-established model of remote storage for distributed file systems. Instead, by colocating computation with PMM storage, we can provide applications with much higher IO performance, sub-second application failover, and strong consistency. To demonstrate this, we built the Assise distributed file system, based on a persistent, replicated coherence protocol that manages client-local PMM as a linearizable and crash-recoverable cache between applications and slower (and possibly remote) storage. Assise maximizes locality for all file IO by carrying out IO on process-local, socket-local, and client-local PMM whenever possible. Assise minimizes coherence overhead by maintaining consistency at IO operation granularity, rather than at fixed block sizes. We compare Assise to Ceph/BlueStore, NFS, and Octopus on a cluster with Intel Optane DC PMMs and SSDs for common cloud applications and benchmarks, such as LevelDB, Postfix, and FileBench. We find that Assise improves write latency up to 22x, throughput up to 56x, fail-over time up to 103x, and scales up to 6x better than its counterparts, while providing stronger consistency semantics.

UR - http://hdl.handle.net/10754/670570

UR - https://www.usenix.org/conference/osdi20/presentation/anderson

M3 - Conference contribution

SP - 1011

EP - 1027

BT - Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation

ER -

Assise: Performance and Availability via Client-local NVM in a Distributed File System

Abstract

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this