Towards automatic parameter tuning of stream processing systems

Muhammad Bilal, Marco Canini

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

39 Scopus citations

Abstract

Optimizing the performance of big-data streaming applications has become a daunting and time-consuming task: parameters may be tuned from a space of hundreds or even thousands of possible configurations. In this paper, we present a framework for automating parameter tuning for stream-processing systems. Our framework supports standard black-box optimization algorithms as well as a novel gray-box optimization algorithm. We demonstrate the multiple benefits of automated parameter tuning in optimizing three benchmark applications in Apache Storm. Our results show that a hill-climbing algorithm that uses a new heuristic sampling approach based on Latin Hypercube provides the best results. Our gray-box algorithm provides comparable results while being two to five times faster.

Original languageEnglish (US)
Title of host publicationSoCC 2017 - Proceedings of the 2017 Symposium on Cloud Computing
PublisherAssociation for Computing Machinery (ACM)
Pages189-200
Number of pages12
ISBN (Electronic)9781450350280
DOIs
StatePublished - Sep 24 2017
Event2017 Symposium on Cloud Computing, SoCC 2017 - Santa Clara, United States
Duration: Sep 24 2017Sep 27 2017

Publication series

NameSoCC 2017 - Proceedings of the 2017 Symposium on Cloud Computing

Conference

Conference2017 Symposium on Cloud Computing, SoCC 2017
Country/TerritoryUnited States
CitySanta Clara
Period09/24/1709/27/17

Bibliographical note

Publisher Copyright:
© 2017 Association for Computing Machinery.

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computational Theory and Mathematics

Fingerprint

Dive into the research topics of 'Towards automatic parameter tuning of stream processing systems'. Together they form a unique fingerprint.

Cite this