You are here: Home / News & Events / DKRZ Tech Talks / Tech Talk: Pack'em ESMs! Tools for (un-)archiving Earth system model data

Tech Talk: Pack'em ESMs! Tools for (un-)archiving Earth system model data

Archiving large amounts of data files via pftp to the HPSS (tape system) is time consuming and takes some manual work. A new set of command line tools called "packems" will be released for all DKRZ users with this Tech Talk. These tools will simplify the archival and retrieval of data files to/from the HPSS – including the creation of tar balls and saving information on where which files are stored.
  • Tech Talk: Pack'em ESMs! Tools for (un-)archiving Earth system model data
  • 2020-10-27T15:15:00+01:00
  • 2020-10-27T16:15:00+01:00
  • Archiving large amounts of data files via pftp to the HPSS (tape system) is time consuming and takes some manual work. A new set of command line tools called "packems" will be released for all DKRZ users with this Tech Talk. These tools will simplify the archival and retrieval of data files to/from the HPSS – including the creation of tar balls and saving information on where which files are stored.
When
Oct 27, 2020 from 03:15 PM to 04:15 PM (Europe/Berlin / UTC100)
Add event to calendar
iCal

Archiving large amounts of data files via pftp to the HPSS (tape system) is time consuming and takes some manual work: (a) users have to aggregate small files in tar balls of 10 GB – 100 GB size; (b) operations with pftp are not handy to automatize. Additionally, the retrieval of small individual files, which have been archived via tar balls in the HPSS, is problematic if users do not remember which files was put into which tar ball. Many users might have experienced these issues.


That's why a new set of command line tools called "packems" is now being made available for all DKRZ users. These tools will simplify the archival and retrieval of data files to/from the HPSS – including the creation of tar balls and saving information on where which files are stored. The "packems" tool set is a joint development of the MPI-M and the DKRZ. It will be presented at the Tech Talk and will be available for usage afterwards on mistral via a module.


Please register by taking the poll at https://terminplaner2.dfn.de/dkrz-tech-talk-packems

Document Actions