You are here: Home / Systems / Mistral / Configuration

Configuration

The Mistral HPC system at DKRZ was installed in two stages.

The Mistral phase 1 system was brought into operation in July 2015 and consists of approx. 1,550 nodes. The compute nodes are housed in bullx B700 DLC (Direct Liquid Cooling) blade systems with two nodes forming one blade. Each node has two sockets, equipped with an Intel Xeon E5-2680 v3 12-core processor (Haswell) sharing 30 MiB L3 cache each. The processor clock-rate is 2.5 GHz.

The Mistral phase 2 system is operational since July 2016 and adds another 1,750 nodes. The phase 2 nodes differ from those of phase 1 in the CPU type only. The new nodes use 2 Intel Xeon CPU E5-2695 v4 (aka Broadwell) CPUs running at 2.1 GHz, and each socket has 18 cores and 45MiB L3 cache. Thus, 24 physical cores per node are available on phase 1 and 36 on phase 2 respectively. Due to active Hyper-Threading, the operating system recognizes two logical CPUs per physical core. The aggregated main memory of the whole system is about 266 TB. The parallel file system Lustre provides 54 PB of usable disk space. The theoretical peak performance of the system is 3.59 PFLOPS/s; the LINPACK performance is about 3.01 PFLOPS/s.

Different kinds of nodes are available to users: 7 login nodes, 5 nodes for interactive data processing and analysis, approx. 3.300 compute nodes for running scientific models, 38 fat memory nodes for pre- and postprocessing of data, and 21 nodes for running advanced visualization or GPGPU applications. The following table lists the specifics of different node types.

Node typeNumber of nodesHostnameProcessors GPGPUsNumber of cores (logical CPUs)Main Memory
login / interactive prepost 7 / 5 mlogin[100-105,108] /
mistralpp[1-5]
2x 12-core Intel Xeon
E5-2680 v3
(Haswell)
@ 2.5GHz


none 24 (48) 256 GB
compute 1404

m[10000-11367], m[11404-11421], m[11560-11577]

2x 12-core Intel Xeon
E5-2680 v3
(Haswell)
@ 2.5GHz

none 24 (48) 64 GB
compute (large memory) 110 m[11368-11403], m[11422,11431], m[11440-11511] 2x 12-core Intel Xeon
E5-2680 v3
(Haswell)
@ 2.5GHz

none 24 (48) 128 GB
compute2 1420

m[20000-21115],
m[21434-21577],
m[21607-21766]

 

2x 18-core Intel Xeon
E5-2695 v4
(Broadwell)
@ 2.1GHz
none 36 (72) 64 GB
compute2 (large memory) 270 m[21116-21385] 2x 18-core Intel Xeon
E5-2695 v4
(Broadwell)
@ 2.1GHz
none 36 (72) 128 GB
compute2 (fat memory) 62 m[21386-21417],
m[21420-21433],
m[21589-21590],
m[21593-21606]
2x 18-core Intel Xeon
E5-2695 v4
(Broadwell)
@ 2.1GHz
none 36 (72) 256 GB
prepost 38 m[11512-11549] 2x 12-core Intel Xeon
E5-2680 v3
(Haswell)
@ 2.5GHz

none 24 (48) 256 GB
visual / gpgpu 12  mg[100-111] 2x 12-core Intel Xeon
E5-2680 v3
(Haswell)
@ 2.5GHz

2x Nvidia Tesla K80,
each with 2x GK210GL
24 (48) 256 GB
" 4 mg[200-203] 2x 18-core Intel Xeon
E5-2695 v4
(Broadwell)
@ 2.1GHz

2x Nvidia Tesla M40
with GM200GL
36 (72) 512 GB
" 1 mg204 2x 18-core Intel Xeon
E5-2695 v4
(Broadwell)
@ 2.1GHz

2x Nvidia Tesla M40  with GM200GL 36 (72) 1024 GB
" 3 mg[205-207] 2x 18-core Intel Xeon
E5-2695 v4
(Broadwell)
@ 2.1GHz

2x Nvidia Quadro M6000 with GM200GL 36 (72) 512 GB
" 1 mg208 2x 18-core Intel Xeon
E5-2695 v4
(Broadwell)
@ 2.1GHz

2x Nvidia Quadro M6000 with GM200GL

36 (72) 1024 GB

 

All compute, pre-/postprocessing, and visualization nodes are integrated in one FDR InfinBand (IB) fabric with three Mellanox SX6536 director switches and fat tree topology with a blocking factor of 1:2:2. The measured bandwidth between two arbitrary compute nodes is 5.9 GByte/s with a latency of 2.7 μs. A scheme of the Infiniband topology is given in the picture below, illustrating the blocking factors depending on which nodes are used for a specific job.

mistral_infiniband_topologie

The Operating System on the Mistral cluster is Red Hat Enterprise Linux release 6.4 (Santiago). The batch system and workload manager is SLURM.

Document Actions