NHR workshop on parallel I/O at DKRZ

Organised co-jointly by Anja Gerbes, National High Performance Computing (NHR), by Dr. Panagiotis Adamidis, DKRZ, and Anna Fuchs as well as Dr. Jannek Squar, University of Hamburg, the event aimed to address key challenges in handling massive datasets, such as those generated by high-resolution climate simulations.

The speakers presented different aspects of I/O and identified problems and possible approaches to solve them. This will be particularly valuable in view of future weather/climate simulation runs at 1 km global resolution, which will require excessive computing power, but also I/O systems.

Discussions focused on efficient methods for parallel I/O on large parallel file systems. Key topics included lossless compression and chunking, selection of appropriate data formats such as HDF5, netCDF and Zarr, and optimisation strategies for I/O in climate models. Participants also looked at post-processing large datasets, monitoring application/file system interactions, and discussed object stores, with a particular focus on reducing time-to-solution.

The workshop concluded on a positive note, with participants agreeing on several promising approaches to improve the current I/O high performance computing environment, paving the way for more efficient climate simulations in the future.

Slides are available at: https://events.dkrz.de/event/62/