Handling Large Numerical Data-Sets: Viability of a Lossy Compressor for CFD-simulations

Patrick Vogler () and Ulrich Rist ()
Additional contact information
Patrick Vogler: High Performance Computing Center
Ulrich Rist: Institut of Aerodynamics and Gasdynamics

A chapter in Sustained Simulation Performance 2019 and 2020, 2021, pp 97-110 from Springer

Abstract: Abstract Over the years, a steady increase in computing power has enabled scientists and engineers to develop increasingly complex applications for machine learning and scientific computing. But while these applications promise to solve some of the most difficult problems we face today, their data hunger also reveals an ever-increasing I/O bottleneck. It is therefore imperative that we develop I/O strategies to better utilize the raw power of our high-performance machines and improve the usability and efficiency of our tools. To achieve this goal, we have developed the BigWhoop compression library based on the JPEG 2000 standard. It enables the efficient and lossy compression of numerical data-sets while minimizing information loss and the introduction of compression artifacts. This paper presents a comparative study using the Taylor-Green Vortex test case to demonstrate the superior compression performance of BigWhoop compared to contemporary solutions. An evaluation of compression-related distortion at high compression ratios is shown to prove its feasibility for both visualization and statistical analysis.

Date: 2021
References: Add references at CitEc
Citations:

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:sprchp:978-3-030-68049-7_7

Ordering information: This item can be ordered from
http://www.springer.com/9783030680497

DOI: 10.1007/978-3-030-68049-7_7

Access Statistics for this chapter

More chapters in Springer Books from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().