You are here: Home / Systems / Mistral / Programming / Python environments

Python environments

Python is increasingly popular in a scientific environment thanks to its wide selection of well maintained modules or packages. On mistral we provide several options ranging from quick usage with a set of predefined modules up to highly customized environments for very specific requirements.

Standard Python installations

These Python installations offer a stable set of installed modules. Existing modules will not be upgraded unless severe security problems need to be addressed. We will install new modules only if their dependencies can be met by already installed versions.

Right now, one of three installations can be activated with the module load command

python/2.7.12  python/2.7-ve0 python/3.5.2

From time to time we start new installations with a recent Python to make more up-to-date modules available.

Anaconda

Anaconda is a python distribution and package manager. Some of the scripts it uses only work in bash. If your login shell is not bash then you have to start a bash shell before working with anaconda.

System wide installation

To load the desired anaconda module you might have to unload/remove conflicting modules first

module unload anaconda2 anaconda3 netcdf_c
module load anaconda3/bleeding_edge

As for the standard Python installations we offer two Anaconda versions based either on Python 2.7 or 3.6. We update these packages on a regular basis in the bleeding edge conda environment. This means that your script can break over time because more recent and potentially incompatible versions of packages have been installed.

This being conda means you can also create your own environments where you have full control over your packages. There are a few things to consider, though.

Conda installs new environments and keeps its package cache in your home directory. This is probably not what you want since your quota on home is very limited. So the first thing to do is to tell conda that environments should be located on work. You have to decide which project's work to use. There can be several locations but new environments are always created in the last addition.

conda config --add envs_dirs '/work/ux9999/u123456/conda_envs'

Then you can start your own environment

conda create -n my_env dask distributed xarray

Activate your environment after all packages got installed

source activate my_env

One drawback of using a system wide conda is that the package cache still resides in your home directory. We don't know of any way to change that. Therefore you should clean the cache after installing many new packages

conda clean --all

You can ignore the errors when conda tries to clean the system package cache.

More information on anaconda can be found on the anaconda documentation page

Individual miniconda installation

Using the system wide conda has some drawbacks like having the package cache always in your home. An individual conda installation gives you full control over every aspect of anaconda while not using much more space than creating your own environment with the system conda.

First download the latest 64-bit bash installer for Linux

Run the installer on mistral like so

bash ./Miniconda3-latest-Linux-x86_64.sh

Be sure to change the installation path to a project work directory, e.g.

/work/ux9999/u123456/miniconda3

You can let the installer add miniconda to your PATH in .bashrc or do it yourself.

Once miniconda's conda is in your PATH, you can start creating your own environments as described for the system wide version above. The package cache is now located in miniconda's installation dirctory on work.

Document Actions