...
main log in
~/.cryosparc/log/run/command_core.log
Code Block title database log less ~/.cryosparc/log/run/command_core.log
supervisor log in
~/.cryosparc/log/run/supervisord.log
Code Block title database log less ~/.cryosparc/log/run/supervisord.log
database log in
~/.cryosparc/log/run/database.log
Code Block title database log less ~/.cryosparc/log/run/database.log
- if there is information that database is "node is not in primary or recovering state" i.e. below
Code Block title cryoSPARC log with error AssertionError: {'code': 500, 'data': None, 'message': "OtherError: node is not in primary or recovering state, full error: {'ok': 0.0, 'errmsg': 'node is not in primary or recovering state', 'code': 13436, 'codeName': 'NotMasterOrSecondary'}
then:
Warning title backup of cryoSPARC projects Backup all your projects and data before removing cryoSPARC database. Usually it should not be a problem to import them without backup but you should backup projects just in case. You have to remember that each cryoSPARC database start naming projects from P1, therefore it is best to backup projects to i.e.
$PLG_GROUPS_STORAGE/<your-plggteam>
space.you have to move current cryoSPARC database
Code Block title database log mv ~/.cryosparc/cryosparc_database ~/.cryosparc/cryosparc_database-$(date+"%Y-%m-%d")
rerun configuration of cryoSPARC to rebuild database
- import projects that you want working with
cryoSPARC master is consuming resources when no other jobs are running
You could run additional helper script called cryosparc_idle_watcher.sh
to check whether there are any cryoSPARC worker jobs running and finish cryoSPARC master job when there aren't any jobs for some time.
Code Block | ||
---|---|---|
| ||
cryosparc_idle_watcher <timeout> |
cryoSPARC master and workers on cluster differ in versions
After changing cryoSPARC master version (i.e. by changing module that you load) cluster lanes are still pointing to previous version of cryoSPARC workers. It could show as error:
Code Block | ||||
---|---|---|---|---|
| ||||
File "cryosparc_worker/cryosparc_compute/run.py", line 46, in cryosparc_compute.run.main
AssertionError: Version mismatch! Worker version (v3.2.0+210629) and Master version (v3.2.0+210413) are not the same. Please update. |
You have to update cluster lanes.
Start interactive job
Code Block language bash title Interactive job srun -p plgrid-services --nodes=1 --ntasks=1 --time=0-1 --pty bash
Load cryoSPARC environment using module (remember to use correct version of cryoSPARC)
Code Block language bash title Load cryoSPARC environment module add plgrid/apps/cryosparc/<version>
Run script that update cluster lanes
Warning title cryoSPARC master job There should be only one job which run cryoSPARC master per user. Therefore you should stop job with cryoSPARC master before this step.
Code Block language bash title Load cryoSPARC environment cryosparc_lanes_updater
Stop cryoSPARC master
Code Block language bash title run cryoSPARC master cryosparcm stop
End interactive job
Code Block language bash title end interactive job exit