You are here: Home / Systems / Cloud Storage / swift

swift

The python swiftclient contains a command line tool to copy data interactively and inside of scripts to and from the cloud storage. It can be used to up- and download large datasets. It is available on all Linux/Unix systems at DKRZ. Swift-Client is open source software and can be downloaded and installed if required.

Supported Platforms

  • Linux
  • OS X
  • Windows

Usage

Swift is already installed on mistral. Before you can use swift please follow these steps

  1. Load module

     Log into mistral and type
    module load swift
  2. After loading the module you have to generate an authentication token by executing
    swift-token new
    This sets some environment variables used by swift as well as creating two files in your home directory (.swiftenv, .swiftenv.tcl). The next time you load the module your token will be re-used if it is still valid (tokens are valid for 7 days by default).
  3. After creating a new token you have load the module again

    module load swift

Help

To display the most important command line options type

swift help

Please visit http://docs.openstack.org/cli-reference/content/swiftclient_commands.html for full reference.

Listing containers and objects

 List all containers

swift list

List all objects in container <containername>

swift list <containername>

Creating containers

Create a new container <containername> (if it doesn't exist)

swift post <containername>

Uploading data

Uploading named files to container <containername>

 swift upload <containername> file [file2, file3, ...]

Single objects are limited to 5 GB in size. If you want to upload larger files, you have to modify the command

swift upload --segment-size 1000000000 <containername> file [file2, file3, ...]

In this example the files are split into chunks of 1 GB size each and put into a container named <containername>_segments. After uploading all chunks a so-called Manifest file with the original filename is created. If this object is downloaded, the server will stream all chunks together transparent for the user.  Use the --segment-container option to specifiy the container where the chunks are placed:

swift upload --segment-size 1000000000 --segment-container <containername> <containername> file [file2, file3, ...]

This is especially useful when uploading data into the scratch space (dkrz_scratch). In order to put all the data into the scratch, you must set the segment-container to  dkrz_scratch. This prevents that the segments are put in a separate container in your account, which would not allow you to use the space of the scratch. So, in order to use the scratch, use following command:

swift upload --segment-size 1000000000 --segment-container dkrz_scratch dkrz_scratch file [file2, file3, ...]
 

Downloading data

Download all objects of all containers

swift download --all

To download all objects from container <containername>

swift download <containername>

To download the file object <objectname> from container <containername>

swift download <containername> <objectname>

When uploading a file with the swift commandline tool, it preserves the current modified time (at least on Unix-like systems) with a metadata entry. When redownloading such a file, the modified time is used for creating the file on the local filesystem. This can lead to problems in certain directories (e.g. tmp, scratch) where files are automatically deleted if they are older than a certain time. Therefore, there exist an option to ignore the modified time stored on the server, so that fresh timestamps are used:

swift download --ignore-mtime <containername> <objectname>

Deleting data

Delete all objects of all containers (be careful what you are doing, because all files are lost!)

swift delete --all

To delete all objects from container <containername>

swift delete <containername>

To delete object <objectname> from container <containername>

swift delete <containername> <objectname>

Share container via ACL

You can share a container via an Access Control List (ACL). For this purpose, you have to add metadata to the container, containing the access right and the account and user, to whom access should be granted.

If you want to grant read access to your container <container> to the user <user> belonging to the account <account>, you add the read ACL via

swift post -r <account>:<user> <container>

If you want to grant access to all users of account <account>, just skip the user part:

swift post -r <account> <container>

The user can then access the container specifiying your storage url. You must inform the user with your own swift storage account name, which you retrieve via the command:

swift stat

The user then accesses the container via:

swift --os-storage-url "https://swift.dkrz.de/v1/<account>" list <container>

If you want to grant write access, use the -w instead of -r switch (or both).

Create temporary URL

First, you need to retrieve the swift storage account name (not the LDAP name)  and the secret key of your account. Use the stat command for this purpose

swift stat

If the secret temporary URL key does not already exist, create one with:

swift post -m "Temp-URL-Key:<key>"

If you want to create a download url,  which is valid for one day, use the following command

swift tempurl GET 86400 /v1/<account>/<containername>/<objectname> <key>

Prepend the server host https://swift.dkrz.de to the output, and send the url to the user. The user can then use for example curl for the download:

curl "https://swift.dkrz.de/v1/<account>/<containername>/<objectname>?\
temp_url_sig=8530f04a7611f6c860243b553d53a6aa1955c1f95&temp_url_expires=1438776977" > file

You can also use ISO 8601 timestamps instead of UNIX timestamps. E.g.:

swift tempurl GET 2017-07-25 /v1/<account>/<containername>/<objectname> <key>

If you want to allow a user to upload data, use PUT instead of GET

swift tempurl PUT 86400 /v1/<account>/<containername>/<objectname> <key>

If the user uses curl for upload, he must specify the method with the -X switch:

curl "https://swift.dkrz.de/v1/<account>/<containername>/<objectname>?\
temp_url_sig=8530f04a7611f6c860243b553d53a6aa1955c1f95&temp_url_expires=1438776977" \
-XPUT --data-binary @file

You can also create the temporary URL using python (http://docs.openstack.org/developer/swift/api/temporary_url_middleware.html).

Prefix-based temporary URL

You can create one temporary URL which is valid for all object names which start with a common prefix. This is useful for sharing a whole container or pseudofolder (= objects with same prefix in the name):

swift tempurl --prefix-based PUT 86400 /v1/<account>/<containername>/<prefix> <key>

A user who wants to upload data into the pseudofolder must inject the object name (which must start with the same prefix) into the generated URL:

https://swift.dkrz.de/v1/<account>/<containername>/<objectname>?temp_url_sig=8530f04a7611f6c860243b553d53a6aa1955c1f95
&temp_url_expires=1438776977&temp_url_prefix=<prefix>

If you want to share a whole container, you must use an empty prefix.

Installing swiftclient on your local system

swift requires Python 2 (2.6 or later). The recommended method is to use pip (see http://docs.openstack.org/user-guide/content/install_clients.html)

pip install python-swiftclient

You need to generate a token for authentication. We have developed a python script for this purpose. You can download it here. If you do not want to use the script, you need to specify your credentials for each request:

swift -U <account>:<username> -A https://swift.dkrz.de/auth/v1.0 -K <key> stat

Document Actions