High-Performance Computing (HPC)

[hongbo-miao]

High-Performance Computing (HPC)

AWS ParallelCluster

• Tutorial
  • Getting started
    • https://docs.aws.amazon.com/parallelcluster/latest/ug/install-v3-configuring.html
    • https://docs.aws.amazon.com/parallelcluster/latest/ug/tutorials-running-your-first-job-on-version-3.html
  • https://catalog.us-east-1.prod.workshops.aws/workshops/21c996a7-8ec9-42a5-9fd6-00949d151bc2/en-US/01-pre
  • Hybrid HPC environment: https://catalog.us-east-1.prod.workshops.aws/workshops/d431e0b1-9f08-4d82-822e-ea56962b2a0b/en-US/slurm-federation
• Tickets
  • Terraform support: Integrate parallelcluster deployment with Terraform aws/aws-parallelcluster#6092
  • Amazon Linux 2023 support: Feature request: AmazonLinux 2023 (al2023) support aws/aws-parallelcluster#5214
  • price-capacity-optimized support: Feature request: price-capacity-optimized aws/aws-parallelcluster#6058
  • Lmod: Support Lmod aws/aws-parallelcluster#5808
• Config

  • EnableMemoryBasedScheduling

    With EnableMemoryBasedScheduling: true, the Slurm scheduler tracks the amount of memory that each job requires on each node. Then, the Slurm scheduler uses this information to schedule multiple jobs on the same compute node. The total amount of memory that jobs require on a node can't be larger than the available node memory. The scheduler prevents a job from using more memory than what was requested when the job was submitted.
    With EnableMemoryBasedScheduling: false, jobs might compete for memory on a shared node and cause job failures and out-of-memory events.

  • QueueUpdateStrategy

    

  • Efa

    Specifies that Elastic Fabric Adapter (EFA) is enabled. To view the list of EC2 instances that support EFA, see Supported instance types in the Amazon EC2 User Guide for Linux Instances. For more information, see Elastic Fabric Adapter. We recommend that you use a cluster SlurmQueues / Networking / PlacementGroup to minimize latencies between instances.
    The default value is false.

  • GdrSupport

    Starting with AWS ParallelCluster version 3.0.2, this setting has no effect.

HPC job managing and scheduling tools

• Slurm: https://slurm.schedmd.com/documentation.html
  • Tutorial: https://www.c4.ucsf.edu/get-started/hello-world-job.html
• (Old) PBS/TORQUE: https://carc.unm.edu/user-support-2/running-jobs/pbstorque.html

Comparison

• https://www.reddit.com/r/HPC/comments/5go1vr/differencesadvantages_of_slurm_vs_torque/
• https://wiki.gacrc.uga.edu/wiki/Migrating_from_Torque_to_Slurm

[hongbo-miao]

Added by

• feat(aws): add aws parallelcluster and slurm #16260
• perf(aws-parallelcluster): set QueueUpdateStrategy, EnableMemoryBasedScheduling, Efa, PlacementGroup #16319

Added diagram as part of the full architecture.

➜ pcluster create-cluster --cluster-name=hm-hpc-cluster --cluster-configuration=config/hm-hpc-cluster-config.yaml
{
  "cluster": {
    "clusterName": "horizon-hpc-cluster",
    "cloudformationStackStatus": "CREATE_IN_PROGRESS",
    "cloudformationStackArn": "arn:aws:cloudformation:us-west-2:272394222652:stack/hm-hpc-cluster/d8cc5ef0-05b0-11ef-bf6e-0ac24af65aed",
    "region": "us-west-2",
    "version": "3.9.1",
    "clusterStatus": "CREATE_IN_PROGRESS",
    "scheduler": {
      "type": "slurm"
    }
  }
}

➜ pcluster ssh --cluster-name=hm-hpc-cluster

ubuntu@ip-172-31-32-251:~$ sbatch --nodes=3 --partition=spot-queue --constraint="[c7gn-16xlarge*1&c7gn-metal*2]" --wrap="srun jobs/hello.sh"
Submitted batch job 1

ubuntu@ip-172-31-32-251:~$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
                 1 spot-queu     wrap   ubuntu CF       0:06      3 spot-queue-dy-c7gn16xlarge-1,spot-queue-dy-c7gnmetal-[1-2]

ubuntu@ip-172-31-32-251:~$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

ubuntu@ip-172-31-32-251:~$ ls
jobs  slurm-1.out

ubuntu@ip-172-31-32-251:~$ cat slurm-1.out
Hello from spot-queue-dy-c7gn16xlarge-1
Hello from spot-queue-dy-c7gnmetal-1
Hello from spot-queue-dy-c7gnmetal-2