Documentation
Tape Archive - HPSS filesystems (High Performance Storage System)
You can use the data archive HPSS to store newly generated data and to read your legacy data from HLRE1.
(External access is restricted to read-only mode.)
The HPSS filesystems with tape based data migration are meant for data which will be (re)used in ongoing projects. Organize your data into large files (e.g. tar archives).
Do not write many small files. Because all your files
will be written randomly to different tapes you will get problems
reading your data back once they have been removed from the disk cache.
Should be used for long term storage of large files (e.g tar archives) only.
We offer different "classes of service" for your data as follows:
| /hpss/arch/<prj> | reference data (single copy on tape), lifetime: 1 year after project expiration |
preferred file size: 10 GB - 50 GB maximum file size: 500 GB |
| /hpss/double/<prj> | like "arch", but with second copy on a separate tape, project's quota (arch) will be charged with twice the amount of data. If you want to move data from "arch" to "double", please contact DKRZ-Beratung |
preferred file size: 10 GB - 50 GB maximum file size: 500 GB |
| LTA (doku) |
long-term archive for documentation data (with second copy), data can be accessed by all registered DKRZ and CERA users, lifetime: up to 10 years after project expiration |
preferred file size: 10 GB - 50 GB maximum file size: 500 GB |
Please note:
If you try to write a file larger than 500 GB, it might be truncated.
The smallest accounting unit is 1 GB, so charging for each file will be rounded up to the nearest GB.
Information about your project's data in HPSS tape archive
A daily updated file called "_PROJECT.<prj>.file-list.GIGA" containing a list of all files stored in your project is located in directory /hpss/arch/<prj> resp. /hpss/double/<prj> and can be fetched from HPSS via pftp.
To get an overview of your project's consumption of space in HPPS archive use our DKRZ online services.
Access via pftp
pftpYou will find yourself in the directory /hpss/arch und can change to your project directory /hpss/arch/<prj>.
Then you can place files into the tape archive by using the put resp mput command and vice versa get them back by using get/mget.
If you want to login automatically use a ~/.netrc file like the following:
machine tape login <user> password <password>
Remember that this file must not be readable by anyone but the user. Special characters have to be escaped with a backslash.
The pftp command is just a stub at the moment but will be expanded by and by and more options will be added.
Please note, that you should avoid pipes when using pftp.
Example for getting/putting data files from/to HPSS:
blizzard$ cd $SCRATCH blizzard$ pftp Connected to ... ftp> pwd 257 "/hpss/arch" is current directory. ftp> cd <prj>/<mydir> ftp> ls # show files on HPSS directory ... <my_data_file> ... ftp> get <my_data_file> ... ftp> !ls # show files on local machine (blizzard,tornado,etc) ... <my_data_file> ftp> quit 221 Goodbye. blizzard$ ls <my_data_file> blizzard$
