Skip to content

Publish the all-in-one image for arm64 too, joined into one manifest list - #960

Open
seagullinfight2077 wants to merge 2 commits into
CopilotKit:mainfrom
seagullinfight2077:fix/openbot-image-arm64
Open

Publish the all-in-one image for arm64 too, joined into one manifest list#960
seagullinfight2077 wants to merge 2 commits into
CopilotKit:mainfrom
seagullinfight2077:fix/openbot-image-arm64

Conversation

@seagullinfight2077

@seagullinfight2077 seagullinfight2077 commented Nov 3, 2026

Copy link
Copy Markdown

English · 繁體中文 · 简体中文

English

What this changes

ghcr.io/copilotkit/openbot, the image README, prompt.txt, docs/deployment.md and the chart all
start a deployment with, is published for linux/amd64 only. latest, v0.2.0 and v0.1.0 each
have a single amd64 manifest, while the component images published from the same workflow
(openbot-server, openbot-supervisor, …) carry amd64 and arm64. The cause is
publish-release.yml: component-images is a matrix over native amd64/arm64 runners joined by
component-manifests, but the image job runs once on ubuntu-latest with no platforms.

So on an Apple Silicon Mac the image runs under Rosetta or QEMU, and on an arm64 node group
(Graviton, Tau T2A, Ampere) it does not run at all. charts/openbot/README.md documents that last
case as a limitation.

This makes the openbot image follow the same pattern as the components, in the same file:

  • image becomes a matrix over ubuntu-latest (amd64) and ubuntu-24.04-arm (arm64). Each leg
    pushes by digest, untagged, and records its digest as digest-openbot-<arch>. The name is never
    manifest-*, because github-release counts those against the component list. Compression stays
    gzip, for the reason the file already gives. The amd64 leg still reads the unscoped cache CI's
    image job writes on main, and each leg writes only its own openbot-<arch> scope.
  • New image-manifest writes the version, commit and latest tags over both digests with
    docker buildx imagetools create. It reads the list digest back and checks it, and attests the
    list digest. The OIDC/attestation permissions move here from image.
  • github-release takes the openbot digest from image-manifest, so container-images.json pins
    the list. gh attestation verify oci://ghcr.io/copilotkit/openbot:vX.Y.Z resolves the tag to that
    same list digest.
  • docs/releasing.md, the chart README's image note and its EKS example, and CHANGELOG.md now say
    this. Releases up to 0.2.0 stay amd64-only.
  • tests/publish-release.test.ts (new) reads the workflow and asserts the following:
    • Every image the release pushes is built on a native runner of each architecture.
    • Every build leg pushes by digest and writes no tags.
    • The openbot tags are written once over both architectures, and the release pins that list.
    • The manifest job waits for the builds and downloads exactly the artifacts they upload, with the
      permissions it needs.
    • No openbot artifact enters the component manifest count.

Where it runs

None of the questions apply: this changes only the release workflow, docs and a test. No server
state, replica behaviour, listener or schedule changes.

Boundary and audit

Not affected: nothing in the server changes.

Changelog

A section under Unreleased: "The all-in-one image runs natively on arm64".

Proof

Test first.

  • On main the new test fails 3 of its first 5 assertions: no amd64/arm64 matrix on image, the
    build leg writes tags, and there is no manifest job. The other 2 pass.
  • With the change, 6/6 pass.
  • Mutation check on a copy of the workflow: each of 8 breakages is caught by at least one failing
    test.
    • Release digest back to needs.image.
    • arm64 leg removed.
    • Attestation on a single-arch digest.
    • needs: image dropped.
    • A one-character download/upload name mismatch.
    • Upload renamed manifest-*.
    • id-token dropped.
    • Build leg writing tags directly.

Static analysis. zizmor 1.32.0 with .github/zizmor.yml at min-severity: low, as CI runs it:
no findings. actionlint 1.7.12 reports only the existing uses: $/.github/workflows/ci.yml (#320),
which is identical on main.

The publishing mechanism, against a local registry. I built the same commit for both
architectures and pushed each by digest. I then ran the merge step's script, extracted unchanged
from the YAML, with REPOSITORY pointed at the local registry. The results:

  • One OCI index, sha256:0929a58c…. The version, commit and latest tags all resolve to it, and it
    equals the sha256 of the raw manifest.
  • It holds linux/amd64 and linux/arm64 plus both BuildKit attestation manifests, each
    vnd.docker.reference.digest pointing back at its own platform.
  • Pulled from the merged latest and booted the way CI's image job does:
    • arm64: uname -m = aarch64, /api/capabilities answered after 8 s, ~395 MiB idle.
    • amd64: uname -m = x86_64 (under Rosetta), answered after 9 s, ~600 MiB idle.
  • Building arm64 natively on an M4 took 707 s.

Not verified

  • The jobs themselves have not run on GitHub. That covers the ubuntu-24.04-arm leg for this
    Dockerfile, the GHA cache scopes, GHCR, and actions/attest-build-provenance on this list. The
    component jobs already exercise each of these in the same workflow.
  • My local amd64 leg differs from CI in three ways:
    • It ran under Rosetta. BuildKit's bundled QEMU aborts in bunx playwright install (signal 6).
    • It used Docker's own builder.
    • It was pushed with a temporary tag, because that builder does not implement push-by-digest.
      Native runners avoid all three.
  • The release checks reuse CI's image job, which boots amd64 only. The arm64 image is booted by
    the local run above, not by CI.

繁體中文

這個改動做了什麼

README、prompt.txt、docs/deployment.md 與 Helm chart 都從 ghcr.io/copilotkit/openbot 開始部署，但這個映像只發布 linux/amd64：latest、v0.2.0、v0.1.0 都只有一個 amd64 manifest；同一個 workflow 發布的元件映像（openbot-server、openbot-supervisor 等）則有 amd64 與 arm64。原因在 publish-release.yml：component-images 是原生 amd64/arm64 runner 的 matrix，再由 component-manifests 合併；image job 卻只在 ubuntu-latest 跑一次、沒設 platforms。結果 Apple Silicon Mac 只能用 Rosetta 或 QEMU 模擬執行，arm64 節點群（Graviton、Tau T2A、Ampere）則完全跑不起來；charts/openbot/README.md 把後者寫成已知限制。

本改動讓 openbot 映像比照同一檔案裡元件映像的做法：

  • image 改成 ubuntu-latest（amd64）與 ubuntu-24.04-arm（arm64）的 matrix。每條腿以 digest 推送、不打 tag，並把 digest 記成 digest-openbot-<arch>。名稱刻意不用 manifest-*，因為 github-release 會拿這類 artifact 的數量去比對元件清單。壓縮維持 gzip（理由檔案裡已寫）。amd64 腿仍會讀 CI 在 main 上寫的預設 cache，兩條腿各自只寫自己的 openbot-<arch> scope。
  • 新增 image-manifest job：用 docker buildx imagetools create 在兩個 digest 上寫入版本、commit、latest 三個 tag，讀回 list 的 digest 並驗證格式，再對 list digest 做 attestation（OIDC 與 attestation 權限從 image 移到這裡）。
  • github-release 改從 image-manifest 取 openbot 的 digest，所以 container-images.json 釘的是這份 list；gh attestation verify oci://ghcr.io/copilotkit/openbot:vX.Y.Z 會把 tag 解析成同一個 list digest。
  • docs/releasing.md、chart README 的映像說明與 EKS 範例、CHANGELOG.md 同步更新。0.2.0 以前（含）的版本仍只有 amd64。
  • 新增 tests/publish-release.test.ts，讀 workflow 並檢查以下幾點：
    • 發布的每個映像都在各架構的原生 runner 上建置。
    • 每條建置腿都以 digest 推送、不寫 tag。
    • openbot 的 tag 只寫一次、涵蓋兩個架構，且 release 釘的是那份 list。
    • manifest job 會等建置完成、下載的正好是建置上傳的 artifact，權限也齊全。
    • openbot 的 artifact 不會被算進元件 manifest 的數量。

在哪裡執行

以上問題都不適用：只改發布 workflow、文件與一個測試。伺服器狀態、多副本行為、listener 與排程都不變。

邊界與審計

不受影響：伺服器端沒有任何改動。

Changelog

在 Unreleased 下新增一段：「The all-in-one image runs natively on arm64」。

證據

先寫測試。

  • 在 main 上，新測試的前 5 條裡有 3 條在斷言失敗：image 沒有 amd64/arm64 matrix、建置腿直接寫 tag、沒有 manifest job。另外 2 條通過。
  • 套用改動後 6/6 通過。
  • 在 workflow 的副本上做變異測試，8 種破壞都至少有一條測試失敗：
    • release digest 改回 needs.image。
    • 拿掉 arm64 腿。
    • attestation 指向單一架構的 digest。
    • 拿掉 needs: image。
    • 上傳和下載的名稱差一個字元。
    • 上傳名稱改成 manifest-*。
    • 拿掉 id-token。
    • 建置腿直接寫 tag。

靜態分析。 zizmor 1.32.0 用 .github/zizmor.yml、min-severity: low（與 CI 相同）：沒有發現。actionlint 1.7.12 只回報既有的 uses: $/.github/workflows/ci.yml（#320），main 上一模一樣。

用本機 registry 實測發布機制。 我用同一個 commit 建兩個架構，各自以 digest 推送，再從 YAML 原封不動取出 merge 步驟的腳本執行（REPOSITORY 指向本機 registry）。結果：

  • 產生一份 OCI index sha256:0929a58c…。版本、commit、latest 三個 tag 都解析到它，且它等於 raw manifest 的 sha256。
  • 內含 linux/amd64、linux/arm64，以及兩份 BuildKit attestation，各自的 vnd.docker.reference.digest 都指回自己的架構。
  • 從合併後的 latest 拉下來，用 CI image job 的方式啟動：
    • arm64：uname -m = aarch64，8 秒回應 /api/capabilities，閒置約 395 MiB。
    • amd64：uname -m = x86_64（Rosetta），9 秒回應，閒置約 600 MiB。
  • 在 M4 上原生建置 arm64 花了 707 秒。

未驗證

  • 這些 job 還沒在 GitHub 上實際跑過，包括這個 Dockerfile 的 ubuntu-24.04-arm 腿、GHA cache scope、GHCR，以及對這份 list 執行 actions/attest-build-provenance。同一個 workflow 的元件 job 已經用到其中每一項。
  • 本機的 amd64 腿和 CI 有三點不同：
    • 用 Rosetta 執行。BuildKit 自帶的 QEMU 會在 bunx playwright install 時 abort（signal 6）。
    • 用的是 Docker 內建 builder。
    • 用暫時 tag 推送，因為那個 builder 不支援 push-by-digest。
      原生 runner 沒有這三個問題。
  • release 的 checks 沿用 CI 的 image job，它只啟動 amd64。arm64 映像的啟動是由上面的本機實測驗證，不是由 CI。

简体中文

这个改动做了什么

README、prompt.txt、docs/deployment.md 与 Helm chart 都从 ghcr.io/copilotkit/openbot 开始部署，但这个映像只发布 linux/amd64：latest、v0.2.0、v0.1.0 都只有一个 amd64 manifest；同一个 workflow 发布的组件映像（openbot-server、openbot-supervisor 等）则有 amd64 与 arm64。原因在 publish-release.yml：component-images 是原生 amd64/arm64 runner 的 matrix，再由 component-manifests 合并；image job 却只在 ubuntu-latest 跑一次、没设 platforms。结果 Apple Silicon Mac 只能用 Rosetta 或 QEMU 模拟运行，arm64 节点群（Graviton、Tau T2A、Ampere）则完全跑不起来；charts/openbot/README.md 把后者写成已知限制。

本改动让 openbot 映像比照同一文件里组件映像的做法：

  • image 改成 ubuntu-latest（amd64）与 ubuntu-24.04-arm（arm64）的 matrix。每条腿以 digest 推送、不打 tag，并把 digest 记成 digest-openbot-<arch>。名称刻意不用 manifest-*，因为 github-release 会拿这类 artifact 的数量去比对组件清单。压缩维持 gzip（理由文件里已写）。amd64 腿仍会读 CI 在 main 上写的预设 cache，两条腿各自只写自己的 openbot-<arch> scope。
  • 添加 image-manifest job：用 docker buildx imagetools create 在两个 digest 上写入版本、commit、latest 三个 tag，读回 list 的 digest 并验证格式，再对 list digest 做 attestation（OIDC 与 attestation 权限从 image 移到这里）。
  • github-release 改从 image-manifest 取 openbot 的 digest，所以 container-images.json 钉的是这份 list；gh attestation verify oci://ghcr.io/copilotkit/openbot:vX.Y.Z 会把 tag 解析成同一个 list digest。
  • docs/releasing.md、chart README 的映像说明与 EKS 范例、CHANGELOG.md 同步更新。0.2.0 以前（含）的版本仍只有 amd64。
  • 添加 tests/publish-release.test.ts，读 workflow 并检查以下几点：
    • 发布的每个映像都在各架构的原生 runner 上建置。
    • 每条建置腿都以 digest 推送、不写 tag。
    • openbot 的 tag 只写一次、涵盖两个架构，且 release 钉的是那份 list。
    • manifest job 会等建置完成、下载的正好是建置上传的 artifact，权限也齐全。
    • openbot 的 artifact 不会被算进组件 manifest 的数量。

在哪里运行

以上问题都不适用：只改发布 workflow、文档与一个测试。服务器状态、多副本行为、listener 与调度都不变。

边界与审计

不受影响：服务器端没有任何改动。

Changelog

在 Unreleased 下添加一段：「The all-in-one image runs natively on arm64」。

证据

先写测试。

  • 在 main 上，新测试的前 5 条里有 3 条在断言失败：image 没有 amd64/arm64 matrix、建置腿直接写 tag、没有 manifest job。另外 2 条通过。
  • 套用改动后 6/6 通过。
  • 在 workflow 的副本上做变异测试，8 种破坏都至少有一条测试失败：
    • release digest 改回 needs.image。
    • 拿掉 arm64 腿。
    • attestation 指向单一架构的 digest。
    • 拿掉 needs: image。
    • 上传和下载的名称差一个字符。
    • 上传名称改成 manifest-*。
    • 拿掉 id-token。
    • 建置腿直接写 tag。

静态分析。 zizmor 1.32.0 用 .github/zizmor.yml、min-severity: low（与 CI 相同）：没有发现。actionlint 1.7.12 只回报既有的 uses: $/.github/workflows/ci.yml（#320），main 上一模一样。

用本机 registry 实测发布机制。 我用同一个 commit 建两个架构，各自以 digest 推送，再从 YAML 原封不动取出 merge 步骤的脚本运行（REPOSITORY 指向本机 registry）。结果：

  • 产生一份 OCI index sha256:0929a58c…。版本、commit、latest 三个 tag 都解析到它，且它等于 raw manifest 的 sha256。
  • 内含 linux/amd64、linux/arm64，以及两份 BuildKit attestation，各自的 vnd.docker.reference.digest 都指回自己的架构。
  • 从合并后的 latest 拉下来，用 CI image job 的方式启动：
    • arm64：uname -m = aarch64，8 秒回应 /api/capabilities，闲置约 395 MiB。
    • amd64：uname -m = x86_64（Rosetta），9 秒回应，闲置约 600 MiB。
  • 在 M4 上原生建置 arm64 花了 707 秒。

未验证

  • 这些 job 还没在 GitHub 上实际跑过，包括这个 Dockerfile 的 ubuntu-24.04-arm 腿、GHA cache scope、GHCR，以及对这份 list 运行 actions/attest-build-provenance。同一个 workflow 的组件 job 已经用到其中每一项。
  • 本机的 amd64 腿和 CI 有三点不同：
    • 用 Rosetta 运行。BuildKit 自带的 QEMU 会在 bunx playwright install 时 abort（signal 6）。
    • 用的是 Docker 内置 builder。
    • 用暂时 tag 推送，因为那个 builder 不支持 push-by-digest。
      原生 runner 没有这三个问题。
  • release 的 checks 沿用 CI 的 image job，它只启动 amd64。arm64 映像的启动是由上面的本机实测验证，不是由 CI。

🤖 Generated with Claude Code

@seagullinfight2077 @claude
Publish the all-in-one image for arm64 too, joined into one manifest …
5b49989 
…list

`ghcr.io/copilotkit/openbot` was built once on ubuntu-latest with no platforms, so every release
(latest, v0.2.0, v0.1.0) is amd64-only while the component images from the same workflow carry both
architectures. The image job now follows component-images/component-manifests: a native
amd64/arm64 matrix pushing by digest, a new image-manifest job that writes the tags over both and
attests the list digest, and github-release pinning that list. Docs, the chart README and the
changelog say so; tests/publish-release.test.ts guards the workflow's shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@seagullinfight2077

seagullinfight2077 commented Nov 3, 2026

Copy link
Copy Markdown
Author

I ran the new jobs on my fork, to close most of the "Not verified" section above:
https://github.com/seagullinfight2077/OpenBot/actions/runs/41604403682

How the test workflow was made

  • It lives on a fork-only branch and is not part of this PR.
  • A script generates it from this PR's publish-release.yml. It takes image and image-manifest key for key.
  • Only three things differ:
    • needs points at a stub metadata job that supplies a version.
    • The release if is removed.
    • The repository is ghcr.io/seagullinfight2077/openbot-arm64-test.
  • It adds a verify job on each native runner.

Results

  • image arm64 on ubuntu-24.04-arm: built and pushed by digest in about 8 min. image amd64 on ubuntu-latest: about 8.5 min. Both started on a cold cache.
  • image manifest wrote the version, commit and latest tags. All three resolve to one OCI index, sha256:e5f39475…, holding linux/amd64 and linux/arm64 plus both BuildKit attestations. It then ran attest-build-provenance on that list digest.
  • verify, on each architecture's native runner:
    • The tag resolves to the digest that image-manifest output.
    • docker pull of the tag gets the runner's own architecture.
    • The image boots as CI's image job boots it, and /api/capabilities answered after 5 s on both (aarch64, x86_64).
  • gh attestation verify oci://…:v0.0.0-fork-test.1 -R seagullinfight2077/OpenBot passes. The subject is the list digest, with SLSA provenance v1. The same command with -R CopilotKit/OpenBot fails, as it should.

Still not covered

  • The real release path is not covered: metadata/verify/checks, the release label gate, and github-release writing container-images.json. These are unchanged except for which job's digest github-release reads.

@seagullinfight2077 @claude
chore: 合併最新 main,解 publish-release.yml 衝突
46fa74f 
main(CopilotKit#979)把 registry 改成依 repo 自動決定(needs.verify.outputs.image_repository),並加了 build-args。
arm64 矩陣和 image-manifest 都改用同一個 registry 名稱,也加上 build-args;
tests/publish-release.test.ts 跟著改比對這個表示式。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@davidmckayv davidmckayv Awaiting requested review from davidmckayv davidmckayv is a code owner
@guidovizoso guidovizoso Awaiting requested review from guidovizoso guidovizoso is a code owner
@tylerslaton tylerslaton Awaiting requested review from tylerslaton tylerslaton is a code owner
@MikeRyanDev MikeRyanDev Awaiting requested review from MikeRyanDev MikeRyanDev is a code owner
@mxmzb mxmzb Awaiting requested review from mxmzb mxmzb is a code owner
@dustywusty dustywusty Awaiting requested review from dustywusty dustywusty is a code owner
@samjulien samjulien Awaiting requested review from samjulien samjulien is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@seagullinfight2077