Overview

HLRE: Filesystems

General Overview

HLRE users have access to a number of different filesystems.
Each filesystem has a special purpose and is configured to best suit this purpose.
You may want to download our user's guide (pdf English) (pdf German) to the (new) project-orientated data management at DKRZ.
In order to make efficient use of the resources PLEASE use the filesystems as advised below.

/prj/<account-id>/doku	$PRJHOME/doku	long-term archive for documentation data (with second copy), lifetime: up to 10 years after project expiration
/prj/<account-id>/arch	$PRJHOME/arch	reference data (single copy on tape), lifetime: 1 year after project expiration
/prj/<account-id>/work	$PRJHOME/work	frequently accessed (temporary/interim) data located on a primarily disk based area, can be processed without having to access tapes *

* The command quotascr on cross shows the amount of data in the work directories according to the user's group membership(s). As the value is newly calculated at each invokation, this may take quite some time.

There are still storage areas for individual users

/shr/?/<user-id>	$WRKSHR	short term data, 150 GB, will be kept 4 days
/wrk-local/<user-id>.nnnnn	$TMPDIR	local filesytem on hurrikan, just accessible during job execution
/pf/<s>/<user-id>	$HOME (on hurrikan)	4GB, for source code, scripts, etc, dumped regularily (mounted to cross as /hpf )

Individual filesystems

Home directories (/pf/<s>/<user-id> aka /hpf/.. resp /ipf/.. ) are used for program sources, configuration files, etc only. They are of moderate size. Overflow is inhibited by quota limiting the data volume per user. Data and consistency is guaranteed by regular filesystem backups.

Filesystems for short term storage ( $TMPDIR, $WRKSHR ) are used for data used by running programs or for job chains. They are optimized for high transfer rates. Lifetime of data is short. There is no backup.
$TMPDIR is directly attached to the compute node your job is running on. Data files are automatically removed after job completion.
$WRKSHR can be accessed from compute and from frontend servers. Quota is 150 GB, lifetime is at least 4 days.

Filesystems used in "old" - not yet project-oriented - data management

The Filesystem for medium term storage ( $UTF, /utf/<s>/<user-id> ) is meant for data which will be (re)used in ongoing projects. This filesystem is optimized for fast transfer so it can also be used for IO intensive diagnostics and data analysis applications. Files are kept up to one year counted from the CREATION of the file. Organize your data into large files (e.g. tar archives of 1 GB and up). DO NOT write many small files. Because all your files will be written randomly to different tapes you will get problems reading your data back once they have been removed from the disk cache.

Archive Filesystem ( $UT, /ut/<s>/<user-id> ) should be used for long term storage of large files (e.g tar archives) only. DO NOT write many small files into the archive. This reduces performance and your data may be distributed over many different tapes.

-> graphical overview over all filesystems

External access is via ftp (see ..)

Deutsches Klimarechenzentrum

HLRE: Filesystems

General Overview

Individual filesystems