...
For important information and announcements, please follow this page and the messages displayed in the login message.
Access to
...
Ahena
Computing resources on Ares Athena are assigned based on PLGrid computing grants. To perform computations on Ares Athena you need to obtain a computing grant, through the PLGrid Portal (https://portal.plgrid.pl/) and apply for Ares Athena access, through the PLGrid portal (https://aplikacje.plgrid.pl/service/dostep-do-klastra-aresathena-w-osrodku-cyfronet/).
The work on Athena's store is still underway, thus there is no dedicated storage for performing high IO computations. For the time being please use the ramdisk $MEMFS functionality as the scratch space (https://kdm.cyfronet.pl/portal/Prometheus:Podstawy#Przestrze.C5.84_dyskowa_w_pami.C4.99ci_operacyjnej_MEMFS). Additionally, in current setup the long term storage is sources from Ares, thus you need to have a grant with storage resources, to be able to use the group directory storage on Athena. Performing high IO computations on group space is strictly forbidden!
If your grant is active, and you have applied for the service access, the request should be accepted in about half an hour, please report any issues through the helpdesk.
...
Available login nodes:
- ssh <login>@ares<login>@athena.cyfronet.pl
Note that Ares Athena uses PLGrid accounts and grants. Make sure to request the "Ares Athena access" access service in the PLGrid portal.
Ares Athena is built with Infiniband EDR HDR interconnect and nodes of the following specification:
| Partition | Number of nodes | CPU | RAM | Accelerator | plgrid | 532 | 48 cores, Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz | 192GB | plgrid-bigmem | 256 | 48 cores, Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz384GB | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| plgrid-gpu-v100a100 | 948 | 32 cores, Intel(R) Xeon(R) Gold 6242 CPU 128 cores, 2x AMD EPYC 7742 64-Core Processor @ 2.80GHz25 GHz | 384GB1024 GB | 8x Tesla V100-SXM2NVIDIA A100-SXM4-40GB |
Job submission
Ares Athena is using Slurm resource manager, jobs should be submitted to the following partitions:
| Name | Timelimit | Account suffix | Remarks | plgrid | 72h | Standard partition.|
|---|---|---|---|---|---|---|
| plgrid- | long168h | Currently unavailable. Used for jobs with extended runtime. | ||||
| plgrid-testing | 1h | Currently unavailable. High priority, testing jobs, limited to 3 jobs. | ||||
| plgrid-bigmem | 72h | Currently unavailable. For jobs using an extended amount of memory. | ||||
| plgrid-now | 12h | Currently unavailable. The highest priority, interactive jobs, 1 running job at most. | ||||
| gpu-a100 | 48h | -gpu | GPU A100 | plgrid-gpu-v100 | 48h | GPU partition. |
Accounts and computing grants
Ares Athena uses a new scheme of naming Slurm accounts for CPU and GPU computing grants. CPU only grants are named: grantname-cpu, while GPU accounts use GPU computing grants using A100 GPU resources use the grantname-gpu suffix-a100 suffix. Please mind that sbatch -A grantname won't work on its own. You need to add the -cpu or gpu-gpua100 suffix! Available computing grants, with respective account names (allocations), can be viewed by using the hpc-grants command.
Resource allocated on Ares Athena doesn't use normalization, 1 hour of CPU GPU time equals 1 hour spent on a computing core, similarly in the case of GPUsusing a GPU.
Storage
Available storage spaces are described in the following table:
| Location | Location in the filesystem | PurposeDescription | |
|---|---|---|---|
| $HOME | /net/people/plgrid/<login> | Storing own applications, and configuration files | |
/net/pr2/scratch/people/<login> or /net/ascratch/people/<login> | Scratch space is currently unavailable, please use a ramdisk created by the $MEMFS functionality: High-speed storage for short-lived data used in computations. Data older than 30 days can be deleted without notice. Please note that scratch can have two physical locations. It is best to rely on the $SCRATCH environment variable. | ||
| $PLG_GROUPS_STORAGE/<group name> | /net/pr2/projects/plgrid/<group name> | Long-term storage, for data living for the period of computing grant. |
...
This space is provided by using Ares storage. If you need permanent space for data, please apply for storage on the Ares cluster. |
System Utilities
Please use the following commands for interacting with the account and storage management system:
...
Applications and libraries are available through the modules system. Please note that the module structure was flattened and module paths might have changed compared to Prometheus! The list of available modules can be obtained by issuing the command:
module avail
to search for a specific module, please use the "spider" (i.e. search) command:
module spider application_name
the list is searchable by using the '/' key. The specific module can be loaded by the add command:
...
module purge
More information
Ares Athena is following Prometheus' configuration and usage patterns. Prometheus documentation can be found here: https://kdm.cyfronet.pl/portal/Prometheus:Basics