Preliminary access essentials
Disclaimer
Athena is still under development, and even despite our best efforts, Ahena might experience unscheduled outages or even data loss.
Content
Support
Please get in touch with the PLGrid Helpdesk: https://helpdesk.plgrid.pl/ regarding any difficulties in using the cluster.
For important information and announcements, please follow this page and the messages displayed in the login message.
Access to Ahena
Computing resources on Athena are assigned based on PLGrid computing grants. To perform computations on Athena you need to obtain a computing grant, through the PLGrid Portal (https://portal.plgrid.pl/) and apply for Athena access, through the PLGrid portal (https://aplikacje.plgrid.pl/service/dostep-do-klastra-athena-w-osrodku-cyfronet/).
The work on Athena's store is still underway, thus there is no dedicated storage for performing high IO computations. For the time being please use the ramdisk $MEMFS functionality as the scratch space (https://kdm.cyfronet.pl/portal/Prometheus:Podstawy#Przestrze.C5.84_dyskowa_w_pami.C4.99ci_operacyjnej_MEMFS). Additionally, in current setup the long term storage is sources from Ares, thus you need to have a grant with storage resources, to be able to use the group directory storage on Athena. Performing high IO computations on group space is strictly forbidden!
If your grant is active, and you have applied for the service access, the request should be accepted in about half an hour, please report any issues through the helpdesk.
Machine description
Available login nodes:
- ssh <login>@athena.cyfronet.pl
Note that Athena uses PLGrid accounts and grants. Make sure to request the "Athena access" access service in the PLGrid portal.
Athena is built with Infiniband HDR interconnect and nodes of the following specification:
| Partition | Number of nodes | CPU | RAM | Accelerator |
|---|---|---|---|---|
| plgrid-gpu-a100 | 48 | 128 cores, 2x AMD EPYC 7742 64-Core Processor @ 2.25 GHz | 1024 GB | 8x NVIDIA A100-SXM4-40GB |
Job submission
Athena is using Slurm resource manager, jobs should be submitted to the following partitions:
| Name | Timelimit | Account suffix | Remarks |
|---|---|---|---|
| plgrid-gpu-a100 | 48h | -gpu | GPU A100 partition. |
Accounts and computing grants
Athena uses a new scheme of naming Slurm accounts GPU computing grants. GPU computing grants using A100 GPU resources use the grantname-gpu-a100 suffix. Please mind that sbatch -A grantname won't work on its own. You need to add the -gpu-a100 suffix! Available computing grants, with respective account names (allocations), can be viewed by using the hpc-grants command.
Resource allocated on Athena doesn't use normalization, 1 hour of GPU time equals 1 hour spent using a GPU.
Storage
Available storage spaces are described in the following table:
| Location | Location in the filesystem | Description |
|---|---|---|
| $HOME | /net/people/plgrid/<login> | Storing own applications, and configuration files |
Scratch space is currently unavailable, please use a ramdisk created by the $MEMFS functionality: | ||
| $PLG_GROUPS_STORAGE/<group name> | /net/pr2/projects/plgrid/<group name> | Long-term storage, for data living for the period of computing grant. This space is provided by using Ares storage. If you need permanent space for data, please apply for storage on the Ares cluster. |
System Utilities
Please use the following commands for interacting with the account and storage management system:
hpc-grants-shows available grants, resource allocationshpc-fs- shows available storagehpc-jobs- shows currently pending/running jobshpc-jobs-history- shows information about past jobs
Software
Applications and libraries are available through the modules system. Please note that the module structure was flattened and module paths might have changed! The list of available modules can be obtained by issuing the command:
module avail
to search for a specific module, please use the "spider" (i.e. search) command:
module spider application_name
The specific module can be loaded by the add command:
module add openmpi/4.1.1-gcc-11.2.0
and the environment can be purged by:
module purge
More information
Athena is following Prometheus' configuration and usage patterns. Prometheus documentation can be found here: https://kdm.cyfronet.pl/portal/Prometheus:Basics