Contents of this guide

With this guide you will learn about workflows - a mechanism that will allow you for automating and managing the data processing inside your workspace. We will tell you how the workflows are constructed and how to use and edit them. Workflows are similar to regular EPISODES Platform applications, therefore, a basic knowledge of the what are these applications and how they are run is a requisite for using workflows, therefore, if you are not familiar with working with the applications yet, please check Applications Quick Start Guide first.

What is a workflow?

A workflow is a sequence of tasks that processes a set of data. It can, in an automated way perform several actions that would, otherwise, have to be invoked one by one. Within the EPISODES Platform, the tasks composing a workflow are the applications - any set of applications present in your workspace can become a workflow. We will show the usage of the workflow mechanism on an example using the application Ground Motion Prediction Equations: GMPE calculation and two other applications that prepare data to be used in the former. The sequence of the applications used is:

1. Ground Motion Parameters Catalog builder (see the application user guide)
2. Catalog Filter (see the application user guide)
3. Ground Motion Prediction Equations: GMPE calculation (see the application user guide)

Normally, when used in workspace, the three applications have to be managed separately, therefore, to run the last application (Ground Motion Prediction Equations: GMPE calculation), one has to first run the Ground Motion Parameters Catalog builder, then use its results for Catalog Filter, and finally, on the results of the latter, create and run the Ground Motion Prediction Equations: GMPE calculation. The amount of work required for running such a sequence of application grows with the number of the component applications, while when using a workflow, the whole sequence is reduced to one operation. Figure 1 shows a comparison between the aforementioned sequence of applications located in workspace and a workflow created from them.

Figure 1. Comparison of application sequence in workspace and a workflow created from them 

A workflow constructed within the EPISODES Platform assumes that the user (a scientist) experiments with the data and its analyses performed by the applications in workspace, and when finding a combination of data and parameters that suits their research best saves that as a workflow. While a workflow itself is a known concept, the approach presented here is quite different from the most common workflow solutions (see Related publications)

Creating a workflow - step by step

Transforming workspace content into a workflow

A workflow can be created by choosing the Transform to workflow action (see Figure 2) from the file menu on any directory (except from the root directory) in workspace. Note: that an application is also a directory, so you can create a workflow also for an application item in workspace.

Figure 2. Transform to workflow action displayed for the base directory menu

As a result, instead of the previous sub-tree, starting from the initial directory (application GMParametersBuilder in Figure 2), your workspace will contain only single entry marked with "W" (meaning workflow) listing results from all the applications that were included in the workflow (you will learn how to configure which results should be shown and which not further in this guide). For example, invoking the Transform to workflow action on the workspace set-up shown in Figure 2, will produce a result as shown in Figure 3 (workspace tree view) and Figure 4 (view of an open workflow). If there were any files in the base workflow directory (in our example, the GMParametersBuilder directory) that were not used by any of the applications, they will simply be hidden from you, as the workflow would only show the content related to the applications inside. Therefore, if you want to have access to these files when the workflow is created, move them outside of its directory.

Figure 3. Directories from Figure 2 after invoking Transform to workflow action

Note, that some inputs - LGCD_Catalog.mat and LGCD_GM_Catalog_1.mat in Figure 3 are outside of the GMParametersBuilder directory. Files like these - inputs of the applications inside the workflow that are outside of the transformed directory will become workflow inputs - see also the description below and Figure 4.

As with every application or file inside the workspace, a workflow can be showed by clicking on its item in the workspace tree (GMParametersBuilder item in Figure 3). Once the workflow is displayed, you can see the following contents:

• Inputs (marked with (1) in Figure 4) - files that are input to the workflow, grouped by the applications that take the files as inputs.
  • Note, that only inputs that were outside of the transformed directory are editable. In the example we used before, LGCD_Catalog.mat and LGCD_GM_Catalog_1.mat files were outside the GMParametersBuilder directory. Other inputs are hidden as, if they were inside the directory, it meant that they were passed from one application to another. You can modify them after expanding the workflow back to directory.
• Forms (marked with (2) in Figure 4) - forms of the applications inside the workflow, again, grouped by the application. By default, forms for all applications are displayed, but they can later be hidden by editing the workflow.
• Controls for running the workflow (marked with (3) in Figure 4) - controls related to running and filling forms, similar to those that are displayed for a regular applications, but this time, they apply to the whole workflow - i.e. the Run button will execute all applications inside the workflow in the correct order, and the Save button will save all the forms, etc.
• Status (marked with (4) in Figure 4) showing the overall workflow status and individual application statuses.
• Outputs (marked with (5) in Figure 4) - outputs of the applications inside the workflow, grouped by the application. By default, outputs of all applications are visible in this section, but you can choose which outputs are visible in the workflow when editing the workflow.

The contents are each displayed in the same order as the order in which the applications were added in the source directory.

Figure 4. View of an open workflow 

Note, that creating (transforming a directory) a workflow does not change the logical structure of applications or files inside, therefore, this operation can be fully undone by expanding the workflow.

Editing the workflow

Editing workflow will allow you to customize which forms and outputs are visible with the workflow view (Figure 4). Note, that not all the features of a workflow are editable, as some of them depend on the structure of the underlying directory. For this reason you cannot remove or add applications to workflow, nor change their order. Neither can you change which are the workflow inputs to the workflow, as they are based on the structure of the inputs of the applications inside. 

To edit the workflow, choose the Edit / debug workflow button marked with (6) in Figure 4. This will change the workflow view, into an editable form - a view similar to previous one, but with additional options to control the editable features - each of them is described below.

Workflow description and application names: 

Debugging the workflow

TODO

Deleting / Expanding the workflow

Once a workflow is created, you have only limited options of reorganization of the applications inside and data flow between them (options described in section Editing the workflow). If, after creating it, you see that the workflow should have different inputs, or the flow of data between the applications inside should be organized differently, you can always expand it with the Expand workflow action accessible from the workspace item menu (see Figure XX). This operation will return the workspace to the state before creating the workflow (in our example, the workspace tree will come back to the situation from Figure 2). You can then rearrange the individual applications, add or delete them, and, when ready, create the workflow anew. Creating and expanding workflows is a light operation, therefore, you can create and expand workflows as you choose.

Figure XX. Expanding a workflow

Related publications

Makuch, M., M. Malawski, J. Kocot, and T. Szepieniec (2020) Applying workflows to scientific projects represented in file system directory tree. In: 2020 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS), pp. 25-32, https://doi.org/10.1109/WORKS51914.2020.00009

The workflow view combines all application forms inside the merged directory. You can create a customized workflow view which shows you:

• Inputs and form applications inside workflow (you can customize which forms are visible)

• Running controls and combined status, which gives you the ability to execute all applications inside the workflow using a single RUN button - the system will execute all applications in the correct order. You can also monitor individual application statuses, save their forms or enable autorun for the entire workflow.

• Outputs form applications and visualizations inside the workflow. You can customize which outputs are visible to hide irrelevant intermediate results.

3. Editing workflow

In the edit view you can customize:

1. Workflow description and application names: 
2. Which forms are visible. You can show/hide the form using + or - buttons:
3. Which outputs are visible. You can show/hide the form using + or - buttons:

To save changes click SAVE WORKFLOW TEMPLATE button:

4. Copying and sharing

Workflow can be easily copied and shared with other IS-EPOS Platform users using COPY and SHARE actions.

Use COPY action and select the target directory:

The workflow will be copied with all internal applications. Workflow template is also copied (visible forms, visible outputs, workflow description and application names).

Use SHARE action and provide the email address of the user you want to share the workflow with:

The user will receive an email with the link to your workflow.

They can add your workflow to their workspace using ADD TO WORKSPACE button. The workflow will be copied with all internal applications. Again workflow template is also copied (visible forms, visible outputs, workflow description and application names).

Nested workflows

TODO

Video version of this guide

This guide is also available as a video: