DKRZ implements new hierarchical data management system for its climate data archive

Following a public tender, NEC was awarded the contract to supply and implement an HSM solution based on NEC hardware and Versity's ScoutAM software.

The HSM system manages DKRZ's entire tape-based long-term storage infrastructure. This includes ten magnetic tape libraries, among them three modern Spectra Logic TFinity libraries. In total, the libraries are equipped 130 LTO drives and can accommodate up to 100,000 LTO magnetic tape cartridges.

The new system serves as the central input/output interface for the tape archive. The new infrastructure is expected to significantly improve aggregate data throughput, enabling climate data to be transferred between the high-performance computing system and the archive at speeds of up to 30 Gigabytes per second. A combined SSD/hard disk cache with a total capacity of 8.2 Petabytes also ensures that peak loads are absorbed and that a portion of the archived data can be retrieved directly—that is, without slow tape accesses.

Currently, 254 Petabytes of simulation data are stored in the archive, distributed across approximately 64 million files. The annual growth rate is currently about 30 Petabytes. This makes it one of the world’s largest archives for climate simulation data.

The hardware for the new system was already delivered in early April. Following installation and technical testing, the metadata of the climate data archive were migrated from the previously used StrongLink HSM system to the new Versity platform. DKRZ users are expected to be able to access the new system starting 10 July.

The Versity software used at the DKRZ is already being utilized by partner institutions for the operation of large magnetic tape archives in Germany by institutions like the Zuse Institute Berlin, the GFZ Helmholtz Center for Geosciences, and the Helmholtz Center Berlin, as well as by international institutions like NASA. Its implementation at the DKRZ thus creates opportunities for exchanging experiences and jointly advancing best practices for archive management.

The upgrade of the data archive is part of the project “High-Performance Computing System for Earth System Research (HLRE-5),” that is jointly funded by the Max Planck Society, the Helmholtz Association of German Research Centers, and the City of Hamburg with a total of 45 million Euros. The HPC system, which is also being renewed, is currently in the procurement process and is scheduled for installation in 2027. The new HSM system provides the foundation needed to sustainably manage the expected growth in data volumes from Earth system research over the long term.