[tooling] Add support for the Task (go-task) task runner #3863

maru-ava · 2025-04-06T19:11:15Z

Why this should be merged

This repo has a lot of tooling but previously had no easy entrypoint other than its github action workflows. This complicated discoverability and CI reproduceability.

This change adds support for invoking every command intended to be run in CI via the Task task runner. Since Task is written in golang, it is easily installable without additional dependency.

The intention is to ensure that every development task can be invoked with scripts/run_task.sh [task] and that the logic of task execution - even for tasks primarily intended for usage in CI - be defined in a given task rather than in a github action workflow.

Alternatives considered:

GNU Make is a build system, and has a lot more complexity and features than we need for a task runner.
Just has a similar featureset to Task but since it is written in Rust it isn't possible to go run on a system without it. task also supports parallel execution which may prove useful in the future.

How this was tested

CI, locally for tasks that aren't tested by CI (e.g. build-image)

Need to be documented in RELEASES.md?

N/A

This repo has a lot of tooling but previously had no easy entrypoint other than its github action workflows. This complicated discoverability and CI reproduceability. This change adds support for invoking every command intended to be run in CI via the [Task](https://taskfile.dev) task runner. Since `Task` is written in golang, it is easily installable without additional dependency. Alternatives considered: - [GNU Make](https://www.gnu.org/software/make/) is a build system, and has a lot more complexity and features than we need for a task runner. - [Just](https://github.com/casey/just) has a similar featureset to Task but since it is written in Rust it isn't possible to `go run` on a system without it.

yacovm · 2025-04-06T20:55:11Z

scripts/tests.e2e.sh

@@ -20,10 +20,6 @@ fi
 # the instructions to build non-portable BLST.
 source ./scripts/constants.sh

-# Enable subnet testing by building xsvm


why is this removed?

It was always a bit weird to have this script build xsvm but not avalanchego. But which version to build - with race detection or without? With the introduction of a task runner, these scripts can become more focused and composable. The test-e2e task can run the build and build-xsvm tasks before this script, and the test-e2e-ci task can reuse the build-race and build-xsvm tasks.

yacovm · 2025-04-06T21:28:01Z

flake.nix

@@ -33,6 +33,9 @@
        default = pkgs.mkShell {
          # The Nix packages provided in the environment
          packages = with pkgs; [
+            # Task runner
+            go-task


so we have nix install go-task and go-task install nix?

More like:

./scripts/run_task.sh install-nix (uses go run to execute task to install nix)

nix develop (starts nix shell that includes go-task)

./scripts/run_task.sh [task] (invokes task from the nix shell) and task (invoked directly) are now equivalent

Using the wrapper script simplifies usage in CI since task can always be invoked whether from a nix shell or a regular shell. For a user manually installing task or using a nix shell, they're free to use task directly.

yacovm · 2025-04-07T00:22:17Z

scripts/actionlint.sh

+  [[ -f "$file" ]] || continue
+
+  # Search for scripts/* except for scripts/run_task.sh
+  MATCHES=$(grep -H -n -P "scripts/(?!run_task\.sh)" "$file" || true)


so this essentially ensures we only use scripts/run_task.sh and nothing else in scripts?

Exactly, the intention is for tasks to be the common interface for both CI and local execution.

tsachiherman

looks good. One thing that I'd rather seeing is having the task tool itself living within our repo ( or a clone that we maintain, which can be used across multiple repos )

tsachiherman · 2025-04-07T17:59:07Z

scripts/run_task.sh

+if command -v task > /dev/null 2>&1; then
+  exec task "${@}"
+else
+  go run github.com/go-task/task/v3/cmd/[email protected] "${@}"


using the above version is good, but specifying the hash instead of the tag is even better:

go run github.com/go-task/task/v3/cmd/task@58ab26c "${@}"

since hashes cannot be retroactivly modified.

Tags are as good as hashes because go by default uses a proxy that effectively renders tags immutable.

maru-ava · 2025-04-07T18:34:15Z

looks good. One thing that I'd rather seeing is having the task tool itself living within our repo ( or a clone that we maintain, which can be used across multiple repos )

Why would we want to maintain the tool in our repo? We depend on many other tools that live outside our repo, I'm not sure what makes task special. scripts/run_task.sh ensures that it can be executed even if not installed, and people have the option of manually installing it in their environment or invoking from within a nix dev shell.

yacovm · 2025-04-07T18:55:24Z

looks good. One thing that I'd rather seeing is having the task tool itself living within our repo ( or a clone that we maintain, which can be used across multiple repos )

Why would we want to maintain the tool in our repo? We depend on many other tools that live outside our repo, I'm not sure what makes task special. scripts/run_task.sh ensures that it can be executed even if not installed, and people have the option of manually installing it in their environment or invoking from within a nix dev shell.

I think if we're worried about the task repository compromised, we can always fork it to our organization and then carefully review the upstream commits and upgrade the forked repository.

StephenButtolph · 2025-04-14T16:30:08Z

flake.nix

@@ -1,6 +1,6 @@
 {
  # To use:
-  #  - install nix: https://github.com/DeterminateSystems/nix-installer?tab=readme-ov-file#install-nix
+  #  - install nix: `./scripts/run_task.sh install_nix`


task: Task "install_nix" does not exist

Good catch, fixed.

StephenButtolph

pretty cool

maru-ava added ci This focuses on changes to the CI process tooling Build, test and development tooling labels Apr 6, 2025

maru-ava self-assigned this Apr 6, 2025

maru-ava requested review from meaghanfitzgerald and StephenButtolph as code owners April 6, 2025 19:11

maru-ava added this to avalanchego Apr 6, 2025

github-project-automation bot moved this to Backlog 🗄️ in avalanchego Apr 6, 2025

maru-ava force-pushed the tooling-add-go-task branch from b0bcd19 to e281954 Compare April 6, 2025 19:12

maru-ava moved this from Backlog 🗄️ to In Review 👀 in avalanchego Apr 6, 2025

maru-ava force-pushed the tooling-add-go-task branch from e281954 to 6775db8 Compare April 6, 2025 19:22

maru-ava force-pushed the tooling-add-go-task branch from 6775db8 to a8f1281 Compare April 6, 2025 19:36

yacovm reviewed Apr 6, 2025

View reviewed changes

maru-ava requested a review from tsachiherman April 6, 2025 22:31

maru-ava force-pushed the tooling-add-go-task branch 2 times, most recently from fef9233 to c030dcf Compare April 6, 2025 23:39

fixup: Added a lint check to ensure all workflows are using task

cb79efc

maru-ava force-pushed the tooling-add-go-task branch from c030dcf to cb79efc Compare April 6, 2025 23:43

yacovm reviewed Apr 7, 2025

View reviewed changes

maru-ava mentioned this pull request Apr 7, 2025

[tooling] Simplify avalanchego build script #3861

Merged

tsachiherman approved these changes Apr 7, 2025

View reviewed changes

StephenButtolph reviewed Apr 14, 2025

View reviewed changes

fixup: Respond to reviewer feedback

b374cbe

StephenButtolph approved these changes Apr 14, 2025

View reviewed changes

maru-ava enabled auto-merge April 14, 2025 16:36

maru-ava added this pull request to the merge queue Apr 14, 2025

Merged via the queue into master with commit ffdd6d0 Apr 14, 2025
24 checks passed

maru-ava deleted the tooling-add-go-task branch April 14, 2025 17:04

github-project-automation bot moved this from In Review 👀 to Done ✅ in avalanchego Apr 14, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[tooling] Add support for the Task (go-task) task runner #3863

[tooling] Add support for the Task (go-task) task runner #3863

Uh oh!

maru-ava commented Apr 6, 2025 •

edited

Loading

Uh oh!

yacovm Apr 6, 2025

Uh oh!

maru-ava Apr 6, 2025

Uh oh!

yacovm Apr 6, 2025

Uh oh!

maru-ava Apr 6, 2025 •

edited

Loading

Uh oh!

yacovm Apr 7, 2025

Uh oh!

maru-ava Apr 7, 2025

Uh oh!

tsachiherman left a comment

Uh oh!

tsachiherman Apr 7, 2025

Uh oh!

maru-ava Apr 7, 2025

Uh oh!

maru-ava commented Apr 7, 2025 •

edited

Loading

Uh oh!

yacovm commented Apr 7, 2025

Uh oh!

StephenButtolph Apr 14, 2025

Uh oh!

maru-ava Apr 14, 2025

Uh oh!

StephenButtolph left a comment

Uh oh!

Uh oh!

Uh oh!

[tooling] Add support for the Task (go-task) task runner #3863

[tooling] Add support for the Task (go-task) task runner #3863

Uh oh!

Conversation

maru-ava commented Apr 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why this should be merged

How this was tested

Need to be documented in RELEASES.md?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maru-ava Apr 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tsachiherman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maru-ava commented Apr 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yacovm commented Apr 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

StephenButtolph left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

maru-ava commented Apr 6, 2025 •

edited

Loading

maru-ava Apr 6, 2025 •

edited

Loading

maru-ava commented Apr 7, 2025 •

edited

Loading