Page History
Contents of this guide
This guide will introduce you to creating and managing your own AI models with the EPOS AI platform. You will learn how to store and evaluate your models in the EPOS AI Workbench as well as how to train them using computing infrastructure and, ultimately, how to publish it and use as a Custom Application of the EPISODES Platform. All these steps are further described on the subsequent pages of this documentation:
Children Display |
---|
Running AI models
On Episodes Platform
Status | ||||
---|---|---|---|---|
|
On user local machines or HPC clusters
The AI tools are available as an ordinary Python packages. We recommend for installation to use Anaconda or Mambaforge, but it should be possible to use any other Python installations or distributions.
Mambaforge installation
Mambaforge is a Python distribution based on Conda. This is a preferred way to run scripts and notebooks distributed on our Platform. It is possible to install and run on other python distributions, but we provide support only for Mambaforge/Anaconda.
The installation starts with downloading the Mambaforge binary from the official project site for your platform. Then follow the instructions in the official guide.
We prepared a Conda environment with all the AI tools installed. To create the environment please:
epos-ai-tools.yml
from our repository.Run the installation
Code Block |
---|
mamba env create -f epos-ai-tools.yml |
Then to activate the environment it is necessary to run for each new shell session:
Code Block |
---|
conda activate epos-ai-tools |
To check if you have enabled the environment correctly you should see the name of the environment in the shell prompt.
Available Tools
The installation comes with the following applications:
- gpd_tool - an application for automatic detection of the first arrival time of the P and S waves based on Generalized Seismic Phase Detection with Deep Learning by Ross, Z. E., Meier, M.-A., Hauksson, E., and T. H. Heaton (2018)
- phasenet_tool - an application for automatic detection of the first arrival time of the P and S waves based on PhaseNet: a deep-neural-network-based seismic arrival-time picking method by Zhu, W., & Beroza, G. C. (2019)
and official packages such as:
- Seisbench - an open-source python toolbox for machine learning in seismology,
- ObsPy - an open-source project dedicated to provide a Python framework for processing seismological data,
- Jupyter Notebook and JupyterLab - a web services for interactive computing,
- IPython - a powerful interactive shell,
- PyTorch - an optimized tensor library for deep learning using GPUs and CPUs
Tools for picking P and S waves
Status | ||||
---|---|---|---|---|
|
Training AI models
Our Platform supports the user with training their own weights for predefined models and even to create their own models. In the first case this could help to improve the quality of the model by training in on data coming from a particular seismic networks (see e.g. Johnson et al. 2021).
To start with training we recommend to use the our HPC clusters which provide enough computing power and access to GPU. The prerequisites are following:
We recommend to apply for resources on a cluster Athena or Ares, but it is possible to run the training on other clusters as well. Please apply for the CPU and GPU computing time, as the GPU significantly accelerates the training process.
.
Entry-level AI Solutions
A straightforward approach to incorporating AI into your research is to employ pre-existing AI models and software to execute particular tasks:
Advanced AI Use Cases
A more advanced route involves the retraining of existing AI models or even the conception and development of customized models and software from the ground up:
Environment for training AI models. Please install the epos-ai-train
environment, by downloading epos-ai-train.yml
from our repository and running:
Code Block |
---|
mamba env create -f epos-ai-train.yml |
Activate the environment it. is necessary to run for each new shell session:
Code Block |
---|
conda activate epos-ai-train |
To check if you have enabled the environment correctly you should see the name of the environment in the shell prompt.
Running jobs on an HPC cluster with SLURM
To submit a job on an HPC cluster use the following job script scheme:
Code Block |
---|
#!/bin/bash
## Set you JOB_NAME to to make it easier to see in the job queue
#SBATCH --job-name=JOB_NAME
## Max task execution time (format is HH:MM:SS)
#SBATCH --time=00:15:00
## Name of grant to which resource usage will be charged
#SBATCH --account=GRANT_ID
## Name of partition
#SBATCH --partition=plgrid
## Number of allocated nodes
#SBATCH --nodes=1
## Number of tasks per node (by default this corresponds to the number of cores allocated per node)
#SBATCH --cpus-per-task=1
## Change to sbatch working directory
cd $SLURM_SUBMIT_DIR
## Activate Python environment
source PATH_TO_MAMBAFORGE_INSTALLATION/mambaforge/bin/activate
conda activate epos-ai-train
## Your usual invoke method e.g.
python train.py |
To enable GPU access for the job please add to the top:
Code Block |
---|
#SBATCH --gres=gpu:1 |
where the number :1 states the number of GPU devices, can be changed to higher numbers. To find which partitions provide access to GPU, please run:
Code Block |
---|
sinfo -o '%P || %N || %G' | column -t |
if in the last column is not null then, it indicates that the partition have access to GPU devices. Please setup the correct partition in the job script.
To find more about submitting jobs please see in corresponding documentation for Ares or Athena.
Available Tools
- Seisbench - an open-source python toolbox for machine learning in seismology,
- Weight & Biases SDK - a library to track experiments, version and iterate on datasets, evaluate model performance, reproduce models, and manage your ML workflows end-to-end,
- ObsPy - an open-source project dedicated to provide a Python framework for processing seismological data,
- Jupyter Notebook and JupyterLab - a web services for interactive computing,
- IPython - a powerful interactive shell,
- PyTorch - an optimized tensor library for deep learning using GPUs and CPUs,
- CUDA - provides a development environment for creating high performance GPU-accelerated applications, a backend library for PyTorch
Weights & Biases
Weights & Biases is the machine learning platform for developers to build better models faster. Use W&B's lightweight, interoperable tools to quickly track experiments, version and iterate on datasets, evaluate model performance, reproduce models, visualize results and spot regressions, and share findings with colleagues. The main features are:
- Track, compare, and visualize your ML models,
- Scalable, customizable hyperparameter search and optimization,
- Visualize and compare every experiment,
- Quickly find and re-run previous model checkpoints,
- Model registry and lifecycle management,
- Lightweight dataset and model versioning,
- Collaborative Reporting.
Acknowledgements
The development of the EPOS AI platform and its integration with the EPISODES Platform was partially funded by the EPOS-PL+ project (No POIR.04.02.00-00-C005/19-00), co-financed by the European Union from the funds of the European Regional Development Fund (ERDF).
We want to express our appreciation for the excellent efforts of the Seisbench developers, and we strongly advise citing their publication:
Woollam, J., Münchmeyer, J., Tilmann, F., Rietbrock, A., Lange, D., Bornstein, T., Diehl, T., Giuchi, C., Haslinger, F., Jozinović, D., Michelini, A., Saul, J., & Soto, H. (2022). SeisBench - A Toolbox for Machine Learning in Seismology. in Seismological Research Letters https://doi.org/10.1785/0220210324
We express our gratitude to Weights & Biases for generously granting us an academic license of their web platform service.