What is a workflow?

A workflow is a sequence of tasks that processes a set of data. It can, in an automated way perform several actions that would, otherwise, have to be invoked one by one. Within the EPISODES Platform, the tasks composing a workflow are the applications (see also Applications Quick Start Guide) - any set of applications present in your workspace can become a workflow. We will show the usage of the workflow mechanism on an example using the application Ground Motion Prediction Equations: GMPE calculation and two other applications that prepare data to be used in the former. The sequence of the applications used is:

1. Ground Motion Parameters Catalog builder (see the application user guide)
2. Catalog Filter (see the application user guide)
3. Ground Motion Prediction Equations: GMPE calculation (see the application user guide)

Normally, when used in workspace, the three applications have to be managed separately, therefore, to run the last application (Ground Motion Prediction Equations: GMPE calculation), one has to first run the Ground Motion Parameters Catalog builder, then use its results for Catalog Filter, and finally, on the results of the latter, create and run the Ground Motion Prediction Equations: GMPE calculation. The amount of work required for running such a sequence of application grows with the number of the component applications, while when using a workflow, the whole sequence is reduced to one operation. Figure 1 shows a comparison between the aforementioned sequence of applications located in workspace and a workflow created from them.

Figure 1. Comparison of application sequence in workspace and a workflow created from them 

A workflow constructed within the EPISODES Platform assumes that the user (a scientist) experiments with the data and its analyses performed by the applications in workspace, and when finding a combination of data and parameters that suits their research best saves that as a workflow. While a workflow itself is a known concept, the approach presented here is quite different from the most common workflow solutions (see Related publications)

Creating a workflow - step by step

Merging workspace content into a workflow

A workflow can be created by choosing the Merge workflow action (see Figure 2) from the file menu on any directory (except from the root directory) in workspace. Note: that an application is also a directory, so you can create a workflow also for an application item in workspace.

Figure 2. Merge workflow action

As a result, instead of the previous sub-tree, starting from the initial directory (application GMParametersBuilder in Figure 2), your workspace will contain only single entry marked with "W" (meaning workflow) listing results from all the applications that were included in the workflow (you will learn how to configure which results should be shown and which not further in this guide). For example, invoking the Merge workflow action on the workspace set-up shown in Figure 2, will produce a result as shown in Figure 3 (workspace tree view) and Figure 4 (view of an open workflow). If there were any files in the base workflow directory (in our example, the GMParametersBuilder directory) that were not used by any of the applications, they will simply be hidden from you, as the workflow would only show the content related to the applications inside. Therefore, if you want to have access to these files when the workflow is created, move them outside of its directory.

Figure 3. Directories from Figure 2 after invoking Merge workflow action

Note, that some inputs - LGCD_catalog.mat and LGCD_GMCatalog 343kb.mat in Figure 3 are outside of the GMParametersBuilder directory. Files like these - inputs that are outside the merged directory will become workflow inputs - the description below and Figure 4.

When open (by clicking on the workflow item) - the workflow view shows:

• Inputs - note, that only inputs that were outside of the merged directory are editable. In the example we used before, LGCD_catalog.mat and LGCD_GMCatalog 343kb.mat files were outside the GMParametersBuilder directory. Other inputs are hidden as, if they were inside the directory, it meant that they were passed from one application to another. You can modify them after expanding the workflow back to directory.
• Forms - a form of each of the application that was inside the directory is displayed here. By default all are displayed, but they can later be hidden while editing the workflow.
• Controls for running the workflow - the Run button will execute all applications inside the workflow in the correct order, and the Save button will save all the forms.
• Status - showing the overall workflow status and individual application statuses.
• Outputs - by default outputs of all applications are visible in this section, you can choose which outputs are visible in the workflow when while editing the workflow.

The contents are each displayed in the same order as the order in which the applications were added in the source directory.

Figure 4. View of an open workflow (TODO: wymienic na taki z formularzami)

Note, that creating (merging) a workflow does not change the logical structure of applications or files inside, therefore, this operation can be fully undone by expanding the workflow.

Editing the workflow

Deleting / Expanding the workflow

Once a workflow is created, you have only limited options of reorganization of the applications inside and data flow between them (options described in section Editing the workflow). If, after creating it, you see that the workflow should have different inputs, or the flow of data between the applications inside should be organized differently, you can always expand it with the Expand workflow action accessible from the workspace item menu (see Figure XX). This operation will return the workspace to the state before creating the workflow (in our example, the workspace tree will come back to the situation from Figure 2). You can then rearrange the individual applications, add or delete them, and, when ready, create the workflow anew. Creating and expanding workflows is a light operation, therefore, you can create and expand workflows as you choose.

Figure XX. Expanding a workflow

Related publications

Makuch, M., M. Malawski, J. Kocot, and T. Szepieniec (2020) Applying workflows to scientific projects represented in file system directory tree. In: 2020 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS), pp. 25-32, https://doi.org/10.1109/WORKS51914.2020.00009

The workflow view combines all application forms inside the merged directory. You can create a customized workflow view which shows you:

• Inputs and form applications inside workflow (you can customize which forms are visible)

• Running controls and combined status, which gives you the ability to execute all applications inside the workflow using a single RUN button - the system will execute all applications in the correct order. You can also monitor individual application statuses, save their forms or enable autorun for the entire workflow.

• Outputs form applications and visualizations inside the workflow. You can customize which outputs are visible to hide irrelevant intermediate results.

3. Editing workflow

You can customize which forms and outputs are visible in the workflow view by clicking EDIT / DEBUG WORKFLOW button:

In the edit view you can customize:

1. Workflow description and application names:
2. Which forms are visible. You can show/hide the form using + or - buttons:
3. Which outputs are visible. You can show/hide the form using + or - buttons:

To save changes click SAVE WORKFLOW TEMPLATE button:

4. Copying and sharing

Workflow can be easily copied and shared with other IS-EPOS Platform users using COPY and SHARE actions.

Use COPY action and select the target directory:

The workflow will be copied with all internal applications. Workflow template is also copied (visible forms, visible outputs, workflow description and application names).

Use SHARE action and provide the email address of the user you want to share the workflow with:

The user will receive an email with the link to your workflow.

They can add your workflow to their workspace using ADD TO WORKSPACE button. The workflow will be copied with all internal applications. Again workflow template is also copied (visible forms, visible outputs, workflow description and application names).

Nested workflows

TODO

Video version of this guide

This guide is also available as a video: