Management of Data Flows Between Cloud, HPC and IoT/Edge

Kamil Tokmakov ()
Additional contact information
Kamil Tokmakov: University of Stuttgart, High Performance Computing Center Stuttgart (HLRS)

A chapter in Sustained Simulation Performance 2022, 2024, pp 25-40 from Springer

Abstract: Abstract The components of heterogeneous applications are deployed across various execution platforms and utilise the capabilities of the platforms. As such, one component can utilise HPC resources for better performance in batch computations, while another—Cloud resources, for better scalability and elasticity. Furthermore, there is also a possibility for processing on Edge devices. The usage of such a hybrid setup, where dependent components of the applications are deployed across various platforms, might require flexible and adaptive data transfers from one platform to another. This work presents a data management framework, based on the Apache NiFi dataflow management system and developed in the scope of the SODALITE EU project. This framework enables scalable data transfer between any of GridFTP (a file transfer protocol common in HPC), HTTP, S3-compatible and data streaming (such as MQTT) endpoints.

Date: 2024
References: Add references at CitEc
Citations:

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:sprchp:978-3-031-41073-4_3

Ordering information: This item can be ordered from
http://www.springer.com/9783031410734

DOI: 10.1007/978-3-031-41073-4_3

Access Statistics for this chapter

More chapters in Springer Books from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().