You are here: Home / Systems / Swift object storage

Swift Object Storage

The object storage system hosted at DKRZ offers a secure platform for saving and distributing data on a global scale.

Use Cases

Swift can be used to share data in various ways:

  • Access data from different devices, e.g. notebook, smartphone and supercomputer
  • Share data with colleagues with or without account at DKRZ
  • Replicate data between different sites automatically

Please notice that all files are kept in three copies on different servers, but there is NO backup from which you could restore deleted or damaged files!

Accounts

Access to Swift is provided for each user which is active in at least one project. There are two types of accounts available: User accounts and project accounts. User accounts are accessible only by the user himself whereas project accounts are shared between all project members.

Quota

  • 50 GiB per user
  • 10 TiB per project

Scratch

We provide a scratch space of about 600 TiB in total.  Please be aware that data put into this space will be automatically deleted after 2 weeks!  You find the scratch in a container called dkrz_scratch in your account, but only if your account is used (which means at least one container has already been created).

Interfaces to Swift

 If you use the RESTful API, you first need to obtain a token. For this, you have to make a GET request to https://swift.dkrz.de/auth/v1.0 with the headers X-Auth-User: <account>:<user> and X-Auth-Key: <password>.

How it works

Please be aware that Swift is a so called "object system", working similar to Amazon S3.

The Object Storage system organizes data in a hierarchy, as follows:

  • Account: Represents the top-level of the hierarchy. The account defines a namespace for containers. A container might have the same name in two different accounts.
  • Container: Defines a namespace for objects. An object with the same name in two different containers represents two different objects. You can create any number of containers within an account. In addition to containing objects, you can also use the container to control access to objects by using an access control list (ACL). You cannot store an ACL with individual objects.
  • Object: Stores data content, such as documents, images, and so on. You can also store custom metadata with an object.

It is important to understand that an object storage system is not used like a POSIX filesystem. For example, there is no concept of a directory. If you want to create a hierachy inside a container, you must include the hierarchy name in your object names. For example, if you want to distinguish between 2 hierarchies "a" and "b", you could call the objects for the first hierarchy "/a/object1". "/a/object2", .... and for the second "/b/object1" , "/b/object2", ... .
Furthermore, moving (without copying) an object from one container to another or renaming an object is not possible. Moving/renaming always includes a (potential) expensive copy operation.

Questions

For technical questions regarding DKRZ's object storage system, please contact

Document Actions