How to handle state in the workflow

[josecelano]

We have a workflow that checks if some Gold images have been updated. We could have new Gold images or some of them could be deleted. For every Gold image, we have the corresponding Base image which should be kept in sync. The base image is a copy from the Gold image with a different size and icc profile. The workflow has some actions to do all the small tasks.

This is the current workflow state:

name: Gold Drawings Processing

on:
  pull_request:
    branches: [ main, issue-* ]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2
        with:
          fetch-depth: 0 # To fetch all history for all branches and tags. Needed for git diff.
      - name: Debug env vars
        ...
      - name: Set env (runner)
        ...
      - name: Git diff
        ...
      - name: DVC diff
        id: dvc-diff
        uses: ./.github/actions/dvc-diff
        env:
          DVC_REPO_DIR: "${{ github.workspace }}"
       ...
      - name: Validate filenames (runner)
        if: ${{ !env.ACT }}
        uses: ./.github/actions/validate-filenames
        with:
          filenames: ${{ steps.dvc-diff.outputs.diff }}
        ...
      - name: Validate Gold images folder (runner)
        if: ${{ !env.ACT }}
        uses: ./.github/actions/validate-gold-image-folder
        with:
          filepaths: ${{ steps.dvc-diff.outputs.diff }}

For the time being, we see two kinds of actions:

1. Generic actions
2. Business domain actions

The first ones are actions like dvc-diff, image-resize, etcetera. The second type are the ones that are specific for the media library: validate-filename, validate-folder.

Every action has its inputs and outputs and in most cases, inputs are obtained from the previous step outputs. This is a sample output for the dvc-diff action:

{
   "added":[
      {
         "path":"data/000001/42/000001-42.600.2.tif"
      }
   ],
   "deleted":[
   ],
   "modified":[
   ],
   "renamed":[
   ],
   "not_in_cache":[
   ],
}

We use that output as an input for the validate-filename action.

We have been discussing how we should connect input/ouputs and what format we should use.

Regarding the format, we have been considering JSON.

Prons

• To easily parse and add, modify or extract data

Cons

• For some simple actions (list of arguments, ...) could be an extra step because you need to build the input data structure.

Regarding how to make the data flow in the pipeline we were talking about different approaches:

1.- Totally independent actions.
2.- Long-term process manager.

1.- Totally independent actions

• Every action does only one thing and it does not know other actions.
• We define the input only with the minimum information we need to execute the task. For example, if we have an action to resize images, we only need the list of images to resize and where you want to write the outputs files.
• In our previous dvc-diff output example we do not need the list of images that are not in the dvc cache on the next step.
• If we follow this approach we need to decide who is going to manipulate input or outputs in order to get just the data you need. In our current implementation, the validate-filename action is taking the full version of the dvc- diff output and extracting the list of added/modified/renamed files. IF we want this action to be totally independent we should remove that code from that action, and maybe add a new action in the middle to transform the output into the next input.

2.- Long-term process manager

@da2ce7 proposed an alternative approach. We can consider the whole workflow as a long-term process (kind of saga). WE have to do a lot of small steps/tasks in order to complete/finish the workflow. Every action it's going to get data from the current state and modify the current state. For example:

• At the beginning we have an empty state.
• The dvc-diff output it's going to be the new next state. The new state contains the list of changed files (in dvc).
• The next action (validate-filename) is going to use the current state as its input. It's going to get data from it and add more data to the state. For example:

{
   "dvc-diff":{
      "added":[
         {
            "path":"data/000001/42/000001-42.600.2.tif"
         }
      ],
      "deleted":[
      ],
      "modified":[
      ],
      "renamed":[
      ],
      "not_in_cache":[
      ]
   },
   "validated-files":{
      "valid":[
         {
            "path":"data/000001/42/000001-42.600.2.tif"
         }
      ],
      "invalid":[
      ],
   },
}

That's only an example. The idea is every action is going to get this data and add or modify it. This pattern could be similar to Flux Pattern. Instead of passing only the data every action needs, the action can use the full state and generate a new version.

Prons

• We do not need intermediary steps to convert/transform json dtos. Each action is responsible for that.
• We can have a dump of the current workflow state. We could even store it in the repo.
• When we trigger one workflow in a repo from another repo we could pass the final workflow state as the event payload to the next workflow in the chain.

Cons

• This makes it harder to write generic GitHub Actions. Actions are somehow coupled to the state structure (unless we use actions composition). There are different solutions to this problem.

Conclusions

• It seems it makes sense to use json to move data around.
• It seems it makes sense to have a central data structure in json with the job (not workflow) state. This json is the database for the job.

We could define for the POC an initial JSON structure. We also have different approaches:

1. Every action uses its own namespace (attribute inside the json).
2. Every action could modify any part of the state and we create an up-front structure for the possible states.
3. There is an initial command to and all the actions add their log to the structure.

@yeraydavidrodriguez is going to re-write images actions with the first implementation of this approach for the POC.