You are here: Home / Systems / HPSS tape archive

HPSS

Tape Archive - HPSS filesystems (High Performance Storage System)

Tape Archive - HPSS filesystems  (High Performance Storage System)


You can use the data archive HPSS to store newly generated data and to read your legacy data from HLRE1.

   (External access is restricted to read-only mode.)

The HPSS  filesystems with tape based data migration are meant for data which will be (re)used in ongoing projects. Organize your data into large files (e.g. tar archives).

Do not write many small files. Because all your files will be written randomly to different tapes you will get problems reading your data back once they have been removed from the disk cache.  Should be used for long term storage of large files (e.g tar archives) only.

We offer different "classes of service" for your data as follows:

 /hpss/arch/<prj> reference data (single copy on tape),
lifetime: 1 year after project expiration
preferred file size:  10 GB - 100 GB
maximum file size: 500 GB
 /hpss/double/<prj> like "arch", but with second copy on a separate tape,
project's quota (arch) will be charged with twice the amount of data.

If you want to move data from "arch" to "double",
please contact  DKRZ-Beratung
preferred file size:  10 GB - 100 GB
maximum file size: 500 GB
 LTA (doku) long-term archive for documentation data (with second copy),
data can be accessed by all registered DKRZ and CERA users,
lifetime:  up to 10 years after project expiration
preferred file size:  10 GB - 100 GB
maximum file size: 500 GB 

Please note:

If you try to write a file larger than 500 GB, it might be truncated.

The smallest accounting unit is 1 GB, so charging for each file will be rounded up to the nearest GB.

 

Information about your project's data in HPSS tape archive

A daily updated file called  "_PROJECT.<prj>.file-list.GIGA"  containing a list of all files stored in your project  is located in directory  /hpss/arch/<prj>  resp.  /hpss/double/<prj>  and can be fetched from HPSS via  pftp.

To get an overview of your project's  consumption of space  in HPPS archive  use our  DKRZ online services.


Access via pftp 

You get access to the tape archive  -  only on mistral and UNIX-machines within ZMAW -  by simply typing
pftp
After changing to your project directory /hpss/arch/<prj>/...   you can place files into the tape archive by using the  put  resp  mput  command and vice versa get them back by using get/mget.

If you want to login automatically use a ~/.netrc file like the following:

machine tape.dkrz.de login <user> password <password>
machine lhp.dkrz.de  login <user> password <password>

Remember that this file must not be readable by anyone but the user. Special characters have to be escaped with a backslash or quoted.

In some configurations an adaption might be necessary. So, if this doesn't work for you,
please contact Beratung

The  pftp  command is just a stub at the moment but will be expanded by and by and more options will be added.
Please note, that you should  avoid pipes when using  pftp.


Example for getting/putting data files from/to HPSS:

mlogin100$ cd $SCRATCH
mlogin100$ pftp
Connected to ...
ftp> cd /hpss/arch/<prj>/path/to/mydir
ftp> ls             # show files on HPSS directory
  ...
my_data_file
  ...
ftp> get my_data_file
  ...
ftp> !ls            # show files on local machine (mistral)
  ...
my_data_file
ftp> quit
221 Goodbye.

mlogin100$ ls
my_data_file
mlogin100$


Recursive operations with pftp

pftp allows certain operations to work recursively. These operations are mget, mput, and mdelete. For instance with recursive mput you can put an entire directory tree with its files into archive

ftp> prompt
Interactive mode off.
ftp> recursive mput mydirectory
257 MKD command successful.
257 MKD command successful.
etc.

Do not upload large numbers of small files. Remember that each file is accounted with at least 1 GB no matter how small.

In a similar fashion, you can also delete an entire directory recursively in archive

ftp> recursive mdel mydirectory
250 DELE command successful.
250 RMD command successful.
250 DELE command successful.

Be careful with non-interactive mode not to delete files you still need. There is no backup for files in archive.

Document Actions