The Merlin HPC Cluster

Merlin6 is the newest generation of the central PSI HPC clusters. In addition to Merlin6, surviving parts of the legacy Merlin5 system are available and maintained on a best effort basis.

The initial installment of Merlin6 was deployed in 2019. The cluster was designed to be extensible regarding the addition of compute nodes and storage. In addition to the main cluster's CPU based resources the system also contains a smaller partition of GPU resources for biology research (Cryo-EM analysis) and machine learning applications.

Hardware Specifications
Service Hardware
  Solution Blade Description

Computing nodes

(Merlin6)

4 x HPE Apollo k6000

Chassis

96 x HPE Apollo XL230K Gen10

(24 blades per chassis)

72 x Two Intel® Xeon® Gold 6152 Scalable Processor @ 2.10GHz (2 x 22 cores per node, HT-enabled, 384GB RAM, NVMe /scratch)

24 x Two Intel(R) Xeon(R) Gold 6240R CPU @ 2.40GHz (2 x 24 cores per node, 18 x 768GB + 6 x 384GB RAM, NVMe / scratch)

HPC Network based on Dual Port Infiniband ConnectX-5 EDR (1 x 100Gbps). Standard network 1 x 10Gbps.

Computing nodes

(Merlin5)

2 x HPE BladeSystem c7000

Chassis

32 x HPE Proliant DL380 Gen8

(16 blades per chassis)

Two Intel® Xeon® Processor E5-2670 @ 2.60GHz (2 x 8 cores, no-HT, 64GB RAM, SAS /scratch)

HPC Network based on Single Port Infiniband ConnectX-3 QDR (1 x 40Gbps). Standard network 1Gbps.

Login nodes

Single Blade

 

2 x HPE Proliant DL380 Gen10

Two Intel® Xeon® Gold 6152 Scalable Processor @ 2.10GHz (2 x 22 cores, HT-enabled, 384GB RAM, NVMe /scratch)

HPC Network based on Dual Port Infiniband ConnectX-5 EDR (2 x 100Gbps). Standard network 2 x 10Gbps.

1 x HPE Proliant DL380 Gen9

Two Intel(R) Xeon(R) CPU E5-2697A v4@ 2.60GHz (2 x 16 cores, HT-enabled, 512GB RAM, SAS /scratch)

HPC Network based on Dual Port Infiniband ConnectIB FDR (1 x 56Gbps). Standard network 1 x 1Gbps.

Storage nodes

Lenovo Distributed Storage Solution for IBM Spectrum Scale

1 x Lenovo DSS G240 Building block

1 x ThinkSystem SR630 (Mgmt node)

2 x ThinkSystem SR650 (IO nodes)

ThinkSystem SR630: Two Intel(R) Xeon(R) Gold 5118 Scalable Processor @ 2.30Ghz (2 x 12 cores, HT-enabled, 96GB RAM

  • Support/management node with xCAT
  • 1 x Dual Port Infiniband ConnectX-5 EDR-100Gbps (low latency network).
  • 1 x Dual Port Infiniband ConnectX-4 EDR-100Gbps (low latency network).
  • Standard network 2 x 10Gbps.

Building block 1:

  • 2 x ThinkSystem SR650: Two Intel(R) Xeon(R) Gold 6142 Scalable Processor @ 2.60Ghz (2 x 16 cores, HT-enabled, 384GB RAM)
    • IO Node
    • 2 x Dual Port Infiniband ConnectX-5 EDR-100Gbps (low latency network).
    • 2 x Dual Port Infiniband ConnectX-4 EDR-100Gbps (low latency network).
    • Standard network 2 x 10Gbps.
    • ThinkSystem RAID 930-8i 2GB Flash PCIe 12Gb Adapter
  • 4 x Lenovo Storage D3284 High Density Expansion Enclosure, each one:
    • Holds 84 x 3.5” hot-swap drive bays in two drawers. Each drawer has three rows of drives, and each row has 14 drives.
    • Each drive bay will contain a 10TB Helium 7.2K NL-SAS HDD.

Other details

HPC Network

Merlin5 Infiniband is based on QDR (40Gbps) and FDR (56Gbps) technologies for MPI communication as well as for storage access. Merlin5 is connected to Merlin6 through FDR (56 Gbps) (MPI)

Merlin6 Infiniband is based on EDR (100Gbps) technology for MPI communication as well as for storage access. Infiniband bandwidth between chassis provide up to 1200Gbps

Network devices

AFS

AFS is available at PSI under the 'psi.ch' domain. This is mounted in the Merlin6 cluster thanks to the Auristor client. AFS contains personal user information as well as the software stack used in the Merlin clusters. AFS is mounted through the standard Ethernet network.

Home directories

Home directories are mounted under the PSI Central NFS service, providing up to 10GB capacity to each each user, with daily snapshots for one week. This is mounted through the standard Ethernet network.

HPC storage

The main storage is based on the IBM's General Parallel FileSystem Spectrum Scale, suitable for HPC environments. This is mounted through the Infiniband network for high performance and low latency.

O.S.

All nodes and servers are running RedHat Enterprise Linux.

Middleware

Remote Desktop

For remote desktop access, the newest login nodes are running NoMachine Terminal Server.

Batch system

The workload manager is based on the Slurm Workload Manager. The Merlin6 Slurm configuration allows running from single core based jobs up to MPI based jobs, allowing to scale up when running over multiple nodes.