What is a workflow?

A workflow is a sequence of tasks that processes a set of data. It can, in an automated way perform several actions that would, otherwise, have to be invoked one by one. Within the EPISODES Platform, the tasks composing a workflow are the applications (see also Applications Quick Start Guide) - any set of applications present in your workspace can become a workflow. We will show the usage of the workflow mechanism on an example using the application Ground Motion Prediction Equations: GMPE calculation and two other applications that prepare data to be used in the former. The sequence of the applications used is:

1. Ground Motion Parameters Catalog builder (see the application user guide)
2. Catalog Filter (see the application user guide)
3. Ground Motion Prediction Equations: GMPE calculation (see the application user guide)

Normally, when used in workspace, the three applications have to be managed separately, therefore, to run the last application (Ground Motion Prediction Equations: GMPE calculation), one has to first run the Ground Motion Parameters Catalog builder, then use its results for Catalog Filter, and finally, on the results of the latter, create and run the Ground Motion Prediction Equations: GMPE calculation. The amount of work required for running such a sequence of application grows with the number of the component applications, while when using a workflow, the whole sequence is reduced to one operation. Figure 1 shows a comparison between the aforementioned sequence of applications located in workspace and a workflow created from them.

Image Modified

Figure 1. Comparison of application sequence in workspace and a workflow created from them 

A workflow constructed within the EPISODES Platform assumes that the user (a scientist) experiments with the data and its analyses performed by the applications in workspace, and when finding a combination of data and parameters that suits their research best saves that as a workflow. While a workflow itself is a known concept, the approach presented here is quite different from the most common workflow solutions (see Related publications)

Creating a workflow - step by step

Merging workspace content into a workflow

A workflow can be created by choosing the Merge workflow action (see Figure 2) from the file menu on any directory (except from the root directory) in workspace. Note: that an application is also a directory, so you can create a workflow also for an application item in workspace.Image Added

Figure 2. Merge workflow action

As a result, instead of the previous sub-tree, starting from the initial directory (application GMParametersBuilder in Figure 2), your workspace will contain only single entry marked with "W" (meaning workflow) listing results from all the applications that were included in the workflow (you will learn how to configure which results should be shown and which not further in this guide). For example, invoking the Merge workflow action on the workspace set-up shown in Figure 2, will produce a result as shown in Figure 3 (workspace tree view) and Figure 4 (view of an open workflow). If there were any files in the base workflow directory (in our example, the GMParametersBuilder directory) that were not used by any of the applications, they will simply be hidden from you, as the workflow would only show the content related to the applications inside. Therefore, if you want to have access to these files when the workflow is created, move them outside of its directory.

Image Added

Figure 3. Directories from Figure 2 after invoking Merge workflow action

• Inputs and form applications inside workflow (you can customize which forms are visible)

Image Removed

• Running controls and combined status, which gives you the ability to execute all applications inside the workflow using a single RUN button - the system will execute all applications in the correct order. You can also monitor individual application statuses, save their forms or enable autorun for the entire workflow.

Image Removed

• Outputs form applications and visualizations inside the workflow. You can customize which outputs are visible to hide irrelevant intermediate results.

Image Removed
2. Creating a workflow

To create a workflow

1. make sure that the given directory contains all applications that should be included inside a workflow;
2. check if all applications have defined inputs and forms, try to execute them;
3. if applications can be executed you can use MERGE WORKFLOW action for the directory which should be merged into the workflow.

Merging workflow does not change the logical structure of applications inside. You can easily undo this operation by clicking EXPAND WORKFLOW action.

Let's take a look at the example set of applications for estimating ground motion prediction equation (GMPE):

1. GMParametersBuilder - Ground Motion Parameters Catalog builder
2. CatalogFilter - Catalog filter
3. GMPEEqsCalc - Ground Motion Prediction Equations: GMPE calculation

All applications are inside GMParametersBuilder directory with the inputs defined as follows:
Image Removed
Note that some inputs (Note, that some inputs - LGCD_catalog.mat and LGCD_GMCatalog 343kb.mat)  in Figure 3 are outside of the GMParametersBuilder directory. The Files like these - inputs that are outside the merged directory will become workflow inputs .
To merge GMParametersBuilder directory to the workflow:

1. Use 3 dots menu near GMParametersBuilder directory, click MERGE WORKFLOW action: Image Removed
2. Accept dialog box message: Image Removed

Image Removed

Image Removed

- the description below and Figure 4.

When open (by clicking on the workflow item) - the workflow view shows:

• Inputs - note, that only inputs that were outside of the merged directory are editable. In

• the example we used before, LGCD_catalog.mat and LGCD_GMCatalog 343kb.mat files were outside the GMParametersBuilder directory. Other inputs are hidden

• as, if they were inside the directory, it meant that they were passed from one application to another. You can modify them

• after expanding the workflow back to directory.

• Forms - a form of each of the application that was inside the directory is displayed here. By default all are displayed, but they can later be hidden while editing the workflow.
• Controls for running

• the workflow

• - the Run button will execute all applications inside the workflow in the correct order

• , and the Save button will save all the forms.
• Status

• - showing the overall workflow status and individual application statuses.
• Outputs

• - by default outputs of all

• applications are visible

• in this section, you can choose which outputs are visible in

• the workflow when while editing the workflow.

The contents are each displayed in the same order as the order in which the applications were added in the source directory.

Image Added

Figure 4. View of an open workflow (TODO: wymienic na taki z formularzami)

Note, that creating (merging) a workflow does not change the logical structure of applications or files inside, therefore, this operation can be fully undone by expanding the workflow.

Anchor
EditingWorkflow EditingWorkflow

Editing the workflow

Anchor
ExpandingWorkflow ExpandingWorkflow

Deleting / Expanding the workflow

Once a workflow is created, you have only limited options of reorganization of the applications inside and data flow between them (options described in section Editing the workflow). If, after creating it, you see that the workflow should have different inputs, or the flow of data between the applications inside should be organized differently, you can always expand it with the Expand workflow action accessible from the workspace item menu (see Figure XX). This operation will return the workspace to the state before creating the workflow (in our example, the workspace tree will come back to the situation from Figure 2). You can then rearrange the individual applications, add or delete them, and, when ready, create the workflow anew. Creating and expanding workflows is a light operation, therefore, you can create and expand workflows as you choose.

Image Added

Figure XX. Expanding a workflow

Anchor
References References

Related publications

Makuch, M., M. Malawski, J. Kocot, and T. Szepieniec (2020) Applying workflows to scientific projects represented in file system directory tree. In: 2020 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS), pp. 25-32, https://doi.org/10.1109/WORKS51914.2020.00009

The workflow view combines all application forms inside the merged directory. You can create a customized workflow view which shows you:

• Inputs and form applications inside workflow (you can customize which forms are visible)

Image Added

• Running controls and combined status, which gives you the ability to execute all applications inside the workflow using a single RUN button - the system will execute all applications in the correct order. You can also monitor individual application statuses, save their forms or enable autorun for the entire workflow.

Image Added

• Outputs form applications and visualizations inside the workflow. You can customize which outputs are visible to hide irrelevant intermediate results.

You can easily delete/expand workflow using EXPAND WORKFLOW action. It will restore the underlying directory so that the workspace structure is the same as before merging a workflow.
Image Removed
If you want to change the workflow structure e.g. add/delete applications or change their inputs you can expand it, modify the content of the directory and merge the directory into workflow again.
Image Removed

Image Added

3. Editing workflow

You can customize which forms and outputs are visible in the workflow view by clicking EDIT / DEBUG WORKFLOW button:

In the edit view you can customize:

1. Workflow description and application names:
2. Which forms are visible. You can show/hide the form using + or - buttons:
3. Which outputs are visible. You can show/hide the form using + or - buttons:

To save changes click SAVE WORKFLOW TEMPLATE button:

Anchor
_xnym8qw8ga4 _xnym8qw8ga4

4. Copying and sharing

Workflow can be easily copied and shared with other IS-EPOS Platform users using COPY and SHARE actions.

Use COPY action and select the target directory:

The workflow will be copied with all internal applications. Workflow template is also copied (visible forms, visible outputs, workflow description and application names).

Use SHARE action and provide the email address of the user you want to share the workflow with:

The user will receive an email with the link to your workflow.

They can add your workflow to their workspace using ADD TO WORKSPACE button. The workflow will be copied with all internal applications. Again workflow template is also copied (visible forms, visible outputs, workflow description and application names).

Nested workflows

TODO

Video version of this guide

This guide is also available as a video: