Configuration

The Mistral HPC system at DKRZ was installed in two stages.

The Mistral phase 1 system was brought into operation in July 2015 and consists of about 1,570 nodes. The compute nodes are housed in bullx B700 DLC (Direct Liquid Cooling) blade systems with two nodes forming one blade. Each node has two sockets, equipped with an Intel Xeon E5-2680 v3 12-core processor (Haswell) sharing 30 MiB L3 cache each. The processor clock-rate is 2.5 GHz.

The Mistral phase 2 system is operational since July 2016 and adds another 1,770 nodes. The phase 2 nodes differ from those of phase 1 in the CPU type. The new nodes use 2 Intel Xeon CPU E5-2695 v4 (aka Broadwell) CPUs running at 2.1 GHz, and each socket has 18 cores and 45MiB L3 cache. Thus, 24 physical cores per node are available on phase 1 and 36 on phase 2 respectively. Due to active Hyper-Threading, the operating system recognizes two logical CPUs per physical core. The aggregated main memory of the whole system is about 266 TB. The parallel file system Lustre provides 54 PB of usable disk space. The theoretical peak performance of the system is 3.59 PFLOPS/s; the LINPACK performance is about 3.01 PFLOPS/s.

Different kinds of nodes are available to users: 7 login nodes, 5 nodes for interactive data processing and analysis, approx. 3.300 compute nodes for running scientific models, 43 fat memory nodes for pre- and postprocessing of data, and 21 nodes for running advanced visualization or GPGPU applications. The following table lists the specifics of different node types.

All compute, pre-/postprocessing, and visualization nodes are integrated in one FDR InfinBand (IB) fabric with three Mellanox SX6536 director switches and fat tree topology with a blocking factor of 1:2:2. The measured bandwidth between two arbitrary compute nodes is 5.9 GByte/s with a latency of 2.7 μs. A scheme of the Infiniband topology is given in the picture below, illustrating the blocking factors depending on which nodes are used for a specific job.

The Operating System on the Mistral cluster is Red Hat Enterprise Linux release 6.4 (Santiago). The batch system and workload manager is SLURM.