ci: multi-arch unit tests + JSON report published to armbian.github.io#872
Merged
igorpecovnik merged 10 commits intomainfrom Apr 18, 2026
Merged
ci: multi-arch unit tests + JSON report published to armbian.github.io#872igorpecovnik merged 10 commits intomainfrom
igorpecovnik merged 10 commits intomainfrom
Conversation
Previous iteration of this branch just swapped `runs-on` from
self-hosted `[docker, X64]` to `ubuntu-24.04-arm`, but the
matrix still hardcoded `ghcr.io/armbian/repository-update:<rel>-amd64`
image tags. armbian/docker-armbian-build publishes those tags as
single-platform manifests (see build-docker-images.yml platforms:
${{ matrix.docker_platform }}), so an arm64 runner pulling an
amd64-only tag fails with 'no matching manifest for linux/arm64/v8'.
Parametrize the matrix on both release and arch. Each test × release
now fans out to two jobs: amd64 on ubuntu-24.04 + arm64 on
ubuntu-24.04-arm, each pulling the matching arch-specific image.
Matrix entries gain two fields:
- arch ("amd64" or "arm64")
- runner (GH-hosted label matching arch)
Downstream: runs-on reads matrix.server.runner; display name and
artifact name carry arch; summary table adds an Arch column.
Lets individual test .conf files opt out of one arch via a comma-separated TESTARCH= allow-list, mirroring the existing colon-separated RELEASE= behaviour. Default is both arches; tests that don't care about both (or rely on e.g. snap amd64-specific shims) can narrow with TESTARCH="amd64" or TESTARCH="arm64". Empty and 'all' both expand to the full arch set. Seed every existing test in tests/*.conf with the default TESTARCH="arm64,amd64" (61 files) placed on the line after RELEASE= so the two filter knobs stay adjacent. No semantic change for the default case — every enabled test still runs on both arches until a maintainer narrows it.
Replace the GH-hosted 'ubuntu-24.04' / 'ubuntu-24.04-arm' pair with the self-hosted label pairs '[docker, X64]' and '[docker, ARM64]'. The runner value in the matrix is a JSON array literal (not a quoted string), injected directly into the JSON without the surrounding quotes — so matrix.server.runner parses back as an array and GHA applies it as the standard 'all these labels required' list form in runs-on.
Add riscv64 to the arches the matrix generator knows about, mapped
to the GH-hosted 'ubuntu-24.04-riscv' runner (self-hosted pool has
no riscv64 host yet). The runner value is emitted as a JSON string
literal for riscv64 vs JSON arrays for amd64/arm64 — runs-on:
${{ matrix.server.runner }} accepts either form.
No change to test .conf defaults: TESTARCH="arm64,amd64" stays,
so existing tests don't auto-pick riscv64. Tests that want riscv64
coverage opt in by editing TESTARCH to e.g.
"arm64,amd64,riscv64" — same pattern as RELEASE.
Prerequisite: ghcr.io/armbian/repository-update:<release>-riscv64
tags must exist. armbian/docker-armbian-build PR #15 (pending)
adds riscv64 to the image-build matrix; until that lands, any
riscv64 unit-test job will fail to pull its container image.
Matrix generator now pipes to 'jq -sc .' (single-line output) and writes to a variable so we can log before handing off to GITHUB_OUTPUT: - compact form rules out any subtle interaction between jq's pretty-printed (multi-line) output and the GITHUB_OUTPUT heredoc delimiter format; - a ::notice:: with the matrix entry count surfaces in the prepare-job log so a future 'gradle skipped with literal template' placeholder (which means the gradle job's matrix != '[]' guard saw '[]') is immediately diagnosable from the summary without having to re-read the loop; - a ::warning:: when the matrix is empty enumerates the most likely causes (no enabled tests, PR didn't touch tests/*.conf and run_all_tests=false, source-time syntax error in a .conf) so the next empty-matrix run lands with actionable output instead of a silent skip.
Two upstream gaps prevent ghcr.io/armbian/repository-update:<rel>-riscv64
images from running unit tests on GH-hosted ubuntu-24.04-riscv:
- image's /sbin/init symlink (systemd-sysv) resolves to a sysvinit
binary that rejects the no-arg background spawn in our
'Initialize systemd' step.
- GH's riscv64 runner doesn't mount Node.js into the container at
/__e/node{N}/, so Node-based actions (actions/checkout, etc.)
can't start inside the container.
Drop riscv64 from this test's TESTARCH. Workflow infrastructure
still recognises riscv64 — any test can opt back in by editing its
TESTARCH once the upstream issues are fixed.
Every test case now writes a per-test JSON blob alongside the
existing workflow output:
{"name": ..., "release": ..., "arch": ...,
"duration": N, "result": "success"|"failure"}
The 'Run unit test' step captures testcase's exit code via an
'if testcase; then ...; else ...; fi' so failures produce a
'failure' result rather than aborting the step before the JSON
is recorded. The step still goes red on failure (explicit
'exit 1' at the end) so the Actions UI shows per-test pass/fail
state accurately.
The merge job (renamed steps):
- aggregates all per-test JSONs into
report/unit-test-results.json with shape
{ generated_at, tests: [sorted by release/arch/name] };
- uploads that file as its own workflow artifact
(unit-test-results, 30-day retention) for downstream consumers;
- derives the Actions-page markdown summary from the JSON
instead of raw echo'd lines (includes a Result column now);
- publishes report/unit-test-results.json to
armbian/armbian.github.io:data/unit-test-results.json via
ARMBIAN_GITHUB_IO_PAT. Gated on the default branch / schedule /
dispatch so PR runs don't write to the public data feed.
Needs the ARMBIAN_GITHUB_IO_PAT repository secret to be a
fine-scoped PAT with write access to armbian/armbian.github.io.
When missing, the publish step emits a ::warning:: and no-ops
rather than failing the workflow.
Match the repository-wide secret name already used elsewhere in the configng CI for cross-repo pushes. No behavioural change.
Contributor
|
Caution Review failedPull request was closed or merged during review WalkthroughThis pull request refactors a GitHub Actions unit test workflow to support multi-architecture testing (arm64 and amd64), replacing fixed architecture mappings with dynamic per-architecture runner and image selections. The workflow now generates test matrix entries across three dimensions (test, release, arch), captures per-test JSON results regardless of outcome status, and produces an aggregated JSON report with markdown table rendering and conditional publication to a separate data repository. In parallel, 53 test configuration files each add an explicit Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes 🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reworks
maintenance-unit-tests.ymlso each test case runs on multiple arches in parallel, with a machine-readable JSON result feed published to the Armbian data pages.Matrix changes
Before: one run per (test × release) against a hardcoded amd64 image tag on
[docker, X64].After: one run per (test × release × arch) with the matching arch-specific image tag and runner.
Runner pool:
[docker, X64]…:<release>-amd64[docker, ARM64]…:<release>-arm64Runner value in the matrix is emitted as a JSON array so
runs-on: ${{ matrix.server.runner }}uses GHA's standard AND-of-labels form.Per-test arch opt-in (
TESTARCH)Every test
.confgains aTESTARCH="arm64,amd64"line (default — fan out to both). Tests that only make sense on one arch can narrow (e.g.TESTARCH="amd64"). Same shape as the existingRELEASE=knob.TESTARCH="all"or empty expands to the full arch set.riscv64
Runner map and arches array are wired for
riscv64→ GH-hostedubuntu-24.04-riscvrunner, but no test opts in yet. Two upstream gaps block it: the repository-update riscv64 image's/sbin/initand GH's riscv64-runner-container Node.js mount. Matrix generator skips the entry gracefully when no test names it — infra stays ready for the day both are fixed.JSON results
Each test writes a per-test blob:
{"name":"XFCE desktop","release":"noble","arch":"amd64","duration":42,"result":"success"}successvsfailurecaptured viaif testcase; then …; else …; fiso a failing test still records its result before the step goes red.The stop job aggregates into
report/unit-test-results.json:{ "generated_at": "2026-04-18T13:00:00Z", "tests": [ {"name":"...","release":"...","arch":"...","duration":42,"result":"success"}, … ] }unit-test-results).Publishing
A new publish step in the stop job pushes
data/unit-test-results.jsontoarmbian/armbian.github.ioon:refs/heads/main), ORschedule,workflow_dispatch,repository_dispatcheventsPR runs are excluded so fork-PRs never write to the public data feed.
Uses the existing
ACCESS_TOKENrepository secret (fine-grained PAT with Contents:Read/Write on armbian.github.io). When missing, the step emits a::warning::and no-ops rather than failing the workflow.Diagnostic logging
Matrix generator logs
::notice::Unit-test matrix has N entrieson every run and, when N=0, a::warning::enumerates the likely causes (no ENABLED tests / changed-files mismatch / source-time syntax error). Compactjq -scoutput avoids any multi-line heredoc interaction.