Aura: A Flexible Dataflow Engine for Scalable Data Processing

Tobias Herb (), Lauritz Thamsen, Thomas Renner and Odej Kao
Additional contact information
Tobias Herb: Technische Universitt Berlin
Lauritz Thamsen: Technische Universitt Berlin
Thomas Renner: Technische Universitt Berlin
Odej Kao: Technische Universitt Berlin

Chapter Chapter 9 in Tools for High Performance Computing 2015, 2016, pp 117-126 from Springer

Abstract: Abstract This paper describes Aura, a parallel dataflow engine for analysis of large-scale datasets on commodity clusters. Aura allows to compose program plans from relational operators and second-order functions, provides automatic program parallelization and optimization, and is a scalable and efficient runtime. Furthermore, Aura provides dedicated support for control flow, allowing advanced analysis programs to be executed as a single dataflow job. This way, it is not necessary to express, for example, data preprocessing, iterative algorithms, or even logic that depends on the outcome of a preceding dataflow as multiple separate jobs. The entire dataflow program is instead handled as one job by the engine, allowing to keep intermediate results in-memory and to consider the entire program during plan optimization to, for example, re-use partitions.

Keywords: Dataflow Engine; Scale Data Processing; Control Flow Operations; Resilient Distributed Datasets (RDDs); Degree Of Parallelism (DOP) (search for similar items in EconPapers)
Date: 2016
References: Add references at CitEc
Citations:

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:sprchp:978-3-319-39589-0_9

Ordering information: This item can be ordered from
http://www.springer.com/9783319395890

DOI: 10.1007/978-3-319-39589-0_9

Access Statistics for this chapter

More chapters in Springer Books from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().