Skip to content

Commit

Permalink
Merge pull request #464 from argonne-lcf/grammar-and-spelling-fixes
Browse files Browse the repository at this point in the history
caught a typo and also made some grammer improvements
  • Loading branch information
felker committed Aug 15, 2024
2 parents 7f87673 + 26aa7ec commit 5dc5047
Showing 1 changed file with 6 additions and 6 deletions.
12 changes: 6 additions & 6 deletions docs/sophia/queueing-and-running-jobs/running-jobs.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
# Running Jobs on Sophia

### Nodes vs Queue
The GPU nodes are NVIDIA DGX A100 nodes and each node contains eight (8) A100 GPUs. The majority of the nodes have the 40 GB A100 models, but two special nodes contain the 80 GB A100 models (see below). You may request either an entire node, or a single GPU based on your job needs. What you will get is determined by the queue you submit to (See Queues section below).
The Sophia nodes are NVIDIA DGX A100 nodes and each node contains eight (8) A100 GPUs. The majority of the nodes have the 40 GB A100 models, but two nodes contain the 80 GB A100 models (see below). You may request resources by node (with 8 GPUs), or by individial GPUs based on your job needs. What you will get is determined by the queue you submit to (See Queues section below).


## <a name="Sophia-Queues"></a>Queues

There are three primary queues:

- `by-gpu`: This is the **default^1^** production queue and is targeted at jobs that can utilize 1-8 GPUs. The number of "chunks" you specify in your `qsub` (i.e. `-l select=4`) will be the number of GPUs you are allocated and they will all be on the same node. Valid values are 1,2,4, or 8 in your select statement. These restrictions ensure you get a sane set of resources (RAM is in the same NUMA node as the cores, the GPU has the minimal hops to the GPU, etc). If you specify a different value your qsub will issue an error and fail.
- `by-node`: This production queue is tarted at jobs that can utilize more than 8 GPUs. The number of "chunks" you specify in your `qsub` (i.e. `-l select=4`) will be the number of Sophia DGX nodes (with 8 GPUs each) you are allocated. Valid values are 1-22 in your select statement. If you request more than 22 your job will never run due to lack of resources.
- `bigmem`: 2 of the nodes have 80GB of RAM per GPU, while the other 22 have 40GB of RAM per GPU (640 GB of aggregate GPU memory per node vs 320 aggregate GPU memory per node). Use this queue to access one of these 2 nodes by specifying ```-q bigmem``` in your script. A max of 1 node (`-l select=1`) can be requested in this queue.
- `by-gpu`: This is the **default^1^** production queue and is targeted at jobs that can utilize 1-8 GPUs. The number of "chunks" you specify in your `qsub` (i.e. `-l select=4`) will be the number of GPUs you are allocated and they will all be on the same node. Valid values are 1,2,4, or 8 in your select statement. These restrictions ensure you get a sane set of resources (RAM is in the same NUMA node as the cores, the GPU has the minimal hops to the GPU, etc). If you specify a different value your `qsub` will issue an error and fail.
- `by-node`: This queue is targeted at jobs that can utilize more than 8 GPUs. The number of "chunks" you specify in your `qsub` (i.e. `-l select=4`) will be the number of Sophia DGX nodes (with 8 GPUs each) you are allocated. Valid values are 1-22 in your select statement. If you request more than 22 your job will never run due to lack of resources.
- `bigmem`: 2 of the nodes have 80GB of RAM per GPU, while the other 22 have 40GB of RAM per GPU (640 GB of aggregate GPU memory per node vs 320 aggregate GPU memory per node). Use this queue to access the 2 nodes with more memory by specifying ```-q bigmem``` in your `qsub`. A max of 1 node (`-l select=1`) can be requested in this queue.

**1:** The default queue is where your job will be submitted if you don't have `-q <queue name>` in your qsub
**1:** The default queue is where your job will be submitted if you don't have `-q <queue name>` in your `qsub`.

Here are the initial queue limits. You may not violate any of these policies when submitting a job:
Here are the initial queue limits. If you violate these limits, your `qsub` will fail.

#### by-gpu queue:
- MinTime is 5 minutes
Expand Down

0 comments on commit 5dc5047

Please sign in to comment.