...
Info | ||
---|---|---|
| ||
Each user should have his/her own their license obtained from https://cryosparc.com/download and apply for membership in |
To get access to cryoSPARC instalation installation at Prometheus cluster
- Apply for apply for membership for membership in
plggcryospar
team in Portal PLGrid and ask for registration in Cyfronet's internal cryoSPARC users' database and a dedicated port for access to cryoSPARC master through through Helpdesk PLGrid. Log in to the Prometheus login node.
Code Block language bash title Log into Prometheus login node ssh <login>@pro.cyfronet.pl
Load cryoSPARC module using command
Code Block language bash title Set cryoSPARC environment module add plgrid/apps/cryosparc
Run cryoSPARC configuration script. It will configure your cryoSPARC environment and create your user in cryoSPARC database and configure two lanes for external jobs -
prometheus-gpu
which is going to use theplgrid-gpu
partition for GPU jobs andprometheus-gpu-v100
forplgrid-gpu-v100
partition. Both lanes are going to useplgrid
partition for CPU-only jobs. As an argument for script pass license id, your e-mail and password (they are going to will be used to login to the cryoSPARC webappweb app), and your first and last name.Code Block language bash title Configure cryoSPARC cryosparc_configuration --license <XXXX> --email <your-email> --password <password> --firstname <Givenname> --lastname <Surname>
Info title Access problems In the case of "
cryosparc_configuration: command not found
" error run in terminalCode Block language bash newgrp plggcryospar
to start a new subshell with permissions of
plggcryospar
team.Info title Access to GPU partitions To use GPUs on the Promehteus cluster, you have to must apply for GPU resources at Portal PLGrid. in your computational grant. At Prometheus, there are two kinds of GPGPUs available - Nvidia Tesla K40 XL in
plgrid-gpu
partition and Nvidia Telsa V100 inplgrid-gpu-v100
partition, and you have to describe in your grant which would you like to use.To check whether you have an access to the partition on the Prometheus login node, run the below command and check whether your PLGrid computational grants are on AllowAccounts list
partition
plgrid-gpu
Code Block language bash scontrol show partition plgrid-gpu | grep Accounts | grep <PLGrid grant name>
partition
plgrid-gpu-v100
Code Block language bash scontrol show partition plgrid-gpu-v100 | grep Accounts | grep <PLGrid grant name>
In case that If you do not have access to one or both of the above partitions, check your PLGrid computational grant details at Portal PLGrid. If your grant lists GPU resources , and access to the required queue or queues is not possible please impossible, don't hesitate to contact Helpdesk at https://helpdesk.plgrid.pl.
- Your cryoSPARC master setup is already done. All succeeding crypoSPARC master instances should should be run in batch jobs.
Warning | ||
---|---|---|
| ||
cryoSPARC master process must not be run on login nodes of the Prometheus cluster. It should be run in using |
...
cryoSPARC master could be started trough through the batch job.
Code Block | ||||
---|---|---|---|---|
| ||||
#!/bin/bash #SBATCH --partition plgrid-services #SBATCH --nodes 1 #SBATCH --ntasks-per-node 4 #SBATCH --mem 10GB #SBATCH --time 14-0 #SBATCH -C localfs #SBATCH --dependency=singleton #SBATCH --job-name cryosparc-master #SBATCH --output cryosparc-master-log-%J.txt #SBATCH --signal=B:2@240 ## Load environment for cryoSPARC module add plgrid/apps/cryosparc ## get tunneling info ipnport=$CRYOSPARC_BASE_PORT ipnip=$(hostname -i) user=$USER ## print tunneling instructions to cryosparc-master-log-<JobID>.txt echo -e " Copy/Paste this in your local terminal to ssh tunnel with remote ----------------------------------------------------------------- ssh -o ServerAliveInterval=300 -N -L $ipnport:$ipnip:$ipnport ${user}@pro.cyfronet.pl ----------------------------------------------------------------- Then open a browser on your local machine to the following address ------------------------------------------------------------------ localhost:$ipnport ------------------------------------------------------------------ " ## start a cryoSPARC master server cryosparcm restart ## loop which keep job running till "scancel --batch --signal=2 <JobID>" by user or automatic kill by SLURM at end of requested walltime source $CRYOSPARC_GLOBAL_ADDITIONAL_FILES_DIR/cryosparc_additional_bash_functions.sh loop-and-stop |
Above The above script is located at $CRYOSPARC_ADDITIONAL_FILES_DIR/cryosparc-master.slurm
. The $CRYOSPARC_ADDITIONAL_FILES_DIR
variable is available when cryoSPARC module is loaded to into the environment. You could copy it into your working folder.
Code Block | ||||
---|---|---|---|---|
| ||||
cp $CRYOSPARC_ADDITIONAL_FILES_DIR/cryosparc-master.slurm . |
...
Submit job
Code Block language bash title job submision sbatch cryosparc-master.slurm
Warning title cryoSPARC master job There should be only one job which run runs cryoSPARC master in
plgrid-servicies
partition per user.Check whether the job was started.
Code Block language bash title jobs status squeue -j <JobID>
Common states of jobs
PD - PENDING
- Job is awaiting resource allocation.R - RUNNING
- Job currently has an allocation and is running.CF - CONFIGURING
- Job has been allocated resources , but are is waiting for them to become ready for use (e.g. booting). On PrometheusCF
state could last for up to 8 minutes in a case when nodes that have been in power save mode.CG - COMPLETING
- Job is in the process of completing. Some processes on some nodes may still be active.
Make a tunnel
In your directory cat job log file:
Code Block language bash title Listing of job's log cat cryosparc-master-log-<JobID>.txt
where `
XXXXXXX
` is your sbatch batch job id which is displayed after you run it f.e. `cat cryosparc-master-log-49145683.txt
`It will show you something like this:
Code Block language bash title Example of job log Copy/Paste this in your local terminal to ssh tunnel with remote ----------------------------------------------------------------- ssh -o ServerAliveInterval=300 -N -L 40100:172.20.68.193:40100 plgusername@pro.cyfronet.pl ----------------------------------------------------------------- Then open a browser on your local machine to the following address ------------------------------------------------------------------ localhost:48511 ------------------------------------------------------------------
Exec in another shell at your local computer given the command to make a tunnel:
Code Block language bash title Tunneling ssh -o ServerAliveInterval=300 -N -L 40100:172.20.68.193:40100 plgusername@pro.cyfronet.pl
- Log into cryoSPARK cryoSPARC web application - open in browser: `
localhost:40100
` To gracefully stop cryoSPARC master in a batch job, send signal "2" using
scancel --batch --signal=2 <JobID>
command.Code Block language bash title Ending master job scancel --batch --signal=2 <JobID>
In case when job won't be stopped by a user through
scancel
it is going to be stopped ended gracefully by sending signal "2" just before maximal time (it is done through#SBATCH --signal=B:2@240
command in the script).
Adding optional lanes
...
Start interactive job using command
Code Block language bash title Interactive job srun -p plgrid-services --nodes=1 --ntasks=1 --mem=5GB --time=0-1 --pty bash
Load cryoSPARC environment using modules
Code Block language bash title Load cryoSPARC environment module add plgrid/apps/cryosparc
Copy cluster config
cluster_info.json
and script templatecluster_script.sh
from$CRYOSPARC_ADDITIONAL_FILES_DIR
directory to your working directoryCode Block language bash title Copy files cp $CRYOSPARC_ADDITIONAL_FILES_DIR/cluster_info.json . cp $CRYOSPARC_ADDITIONAL_FILES_DIR/cluster_script.sh .
- Modify files accordingly
- in config
cluster_info.json
change the name of lane/cluster to avoid overwriting default prometheus* lanes - in
cluster_script.sh
change--time
,--partition
or other parts of the script template accordingly
- in config
Start cryoSPARC master
Warning title cryoSPARC master job There should be only one job which run runs cryoSPARC master per user. Therefore you should stop the job with cryoSPARC master before this step.
Code Block language bash title run cryoSPARC master cryosparcm restart
run command
cryosparcm cluster connect <name-of-cluster-form-cluster_info.json>
to add lane/clusterCode Block language bash title add line cryosparcm cluster connect
- Repeat the above points to create another lane if necessary
Stop cryoSPARC master
Code Block language bash title run cryoSPARC master cryosparcm stop
End interactive job
Code Block language bash title end interactive job exit
Upgrading
...
to a newer version
cryoSPARC is updated frequently with updates and patches (see https://cryosparc.com/updates). For each new version, a new module in plgrid/apps/cryosparc/<version>
is created, which is made default after testing (among other test tests also with standard benchmark). To upgrade to a new version, one has to
- End all running cryoSPARC jobs.
- Update
cryosparc-master.slurm
script (or check that it uses the default cryoSPARC module). - Manually update all lanes created (both those created during configuration and added later). Script described in Known problems and issues section could be used to automate this process.
- Restart cryoSPARC master server.
...