fix(lambda): add jti claim to GitHub App JWTs to prevent concurrent collisions#5056
Merged
Brend-Smits merged 1 commit intoMar 6, 2026
Merged
Conversation
…ollisions When multiple scale-up Lambda invocations run concurrently, the JWT generation via universal-github-app-jwt produces byte-identical tokens because it only includes iat, exp, and iss claims with second-precision timestamps. GitHub rejects duplicate JWTs, returning HTTP 404 on the installation token endpoint, which causes silent batch dropping. Replace the privateKey-based auth with a custom createJwt callback using the @octokit/auth-app first-class API. The callback signs JWTs with node:crypto (zero new dependencies) and includes a randomUUID() jti claim, ensuring every token is unique even within the same second. Fixes github-aws-runners#5025
Brend-Smits
approved these changes
Mar 6, 2026
Contributor
Brend-Smits
left a comment
There was a problem hiding this comment.
Hey!
Thanks for this great contribution 🚀
Tested this in our testing environment and things seem to work well. Okay from me (and @stuartp44) side.
npalm
pushed a commit
that referenced
this pull request
Mar 9, 2026
🤖 I have created a release *beep* *boop* --- ## [7.4.1](v7.4.0...v7.4.1) (2026-03-09) ### Bug Fixes * gracefully handle JIT config failures and terminate unconfigured instance ([#4990](#4990)) ([c171550](c171550)) * **install-runner.sh:** support Debian ([#5027](#5027)) ([7755b7f](7755b7f)) * **lambda:** add jti claim to GitHub App JWTs to prevent concurrent collisions ([#5056](#5056)) ([07bd193](07bd193)), closes [#5025](#5025) * **lambda:** bump @octokit/auth-app from 8.1.2 to 8.2.0 in /lambdas in the octokit group ([#5035](#5035)) ([1c8083e](1c8083e)) * **lambda:** bump axios from 1.13.2 to 1.13.5 in /lambdas ([#5028](#5028)) ([0335e3a](0335e3a)) * **lambda:** bump qs from 6.14.1 to 6.14.2 in /lambdas ([#5032](#5032)) ([6dc97d5](6dc97d5)) * **lambda:** bump rollup from 4.46.2 to 4.59.0 in /lambdas ([#5052](#5052)) ([1e798b1](1e798b1)) * **lambda:** bump the aws group in /lambdas with 7 updates ([#5021](#5021)) ([c3c158d](c3c158d)) * **lambda:** bump the aws-powertools group in /lambdas with 4 updates ([#5022](#5022)) ([e8369cf](e8369cf)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: runners-releaser[bot] <194412594+runners-releaser[bot]@users.noreply.github.com>
Brend-Smits
pushed a commit
that referenced
this pull request
Mar 11, 2026
🤖 I have created a release *beep* *boop* --- ## [7.4.1](v7.4.0...v7.4.1) (2026-03-09) ### Bug Fixes * gracefully handle JIT config failures and terminate unconfigured instance ([#4990](#4990)) ([c171550](c171550)) * **install-runner.sh:** support Debian ([#5027](#5027)) ([7755b7f](7755b7f)) * **lambda:** add jti claim to GitHub App JWTs to prevent concurrent collisions ([#5056](#5056)) ([07bd193](07bd193)), closes [#5025](#5025) * **lambda:** bump @octokit/auth-app from 8.1.2 to 8.2.0 in /lambdas in the octokit group ([#5035](#5035)) ([1c8083e](1c8083e)) * **lambda:** bump axios from 1.13.2 to 1.13.5 in /lambdas ([#5028](#5028)) ([0335e3a](0335e3a)) * **lambda:** bump qs from 6.14.1 to 6.14.2 in /lambdas ([#5032](#5032)) ([6dc97d5](6dc97d5)) * **lambda:** bump rollup from 4.46.2 to 4.59.0 in /lambdas ([#5052](#5052)) ([1e798b1](1e798b1)) * **lambda:** bump the aws group in /lambdas with 7 updates ([#5021](#5021)) ([c3c158d](c3c158d)) * **lambda:** bump the aws-powertools group in /lambdas with 4 updates ([#5022](#5022)) ([e8369cf](e8369cf)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: runners-releaser[bot] <194412594+runners-releaser[bot]@users.noreply.github.com>
Brend-Smits
pushed a commit
that referenced
this pull request
Apr 1, 2026
🤖 I have created a release *beep* *boop* --- ## [7.4.1](v7.4.0...v7.4.1) (2026-03-09) ### Bug Fixes * gracefully handle JIT config failures and terminate unconfigured instance ([#4990](#4990)) ([c171550](c171550)) * **install-runner.sh:** support Debian ([#5027](#5027)) ([7755b7f](7755b7f)) * **lambda:** add jti claim to GitHub App JWTs to prevent concurrent collisions ([#5056](#5056)) ([07bd193](07bd193)), closes [#5025](#5025) * **lambda:** bump @octokit/auth-app from 8.1.2 to 8.2.0 in /lambdas in the octokit group ([#5035](#5035)) ([1c8083e](1c8083e)) * **lambda:** bump axios from 1.13.2 to 1.13.5 in /lambdas ([#5028](#5028)) ([0335e3a](0335e3a)) * **lambda:** bump qs from 6.14.1 to 6.14.2 in /lambdas ([#5032](#5032)) ([6dc97d5](6dc97d5)) * **lambda:** bump rollup from 4.46.2 to 4.59.0 in /lambdas ([#5052](#5052)) ([1e798b1](1e798b1)) * **lambda:** bump the aws group in /lambdas with 7 updates ([#5021](#5021)) ([c3c158d](c3c158d)) * **lambda:** bump the aws-powertools group in /lambdas with 4 updates ([#5022](#5022)) ([e8369cf](e8369cf)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: runners-releaser[bot] <194412594+runners-releaser[bot]@users.noreply.github.com>
LudovicTOURMAN
pushed a commit
to doctolib-lab/terraform-aws-github-runner
that referenced
this pull request
Apr 7, 2026
…ollisions (github-aws-runners#5056) ## Summary Fixes concurrent JWT collisions that cause silent job loss during burst workloads. When multiple scale-up Lambda invocations generate GitHub App JWTs within the same second, `universal-github-app-jwt` produces byte-identical tokens (same `iat`, `exp`, `iss`, no `jti`). GitHub rejects the duplicates, returning HTTP 404 on `POST /app/installations/{id}/access_tokens`, which triggers silent batch dropping. ### Root cause `universal-github-app-jwt` generates JWTs with only `{ iat, exp, iss }` claims. The `iat` uses seconds precision (`Math.floor(Date.now() / 1000)`). With the same App ID and private key, concurrent invocations within the same second produce identical tokens. ### Fix Replace `privateKey`-based auth with a custom `createJwt` callback — a first-class API in `@octokit/auth-app` v8.x that completely bypasses `universal-github-app-jwt`. The callback: - Signs JWTs using `node:crypto.createSign` (zero new dependencies) - Includes a `crypto.randomUUID()` `jti` claim, ensuring every token is unique - Preserves the existing `iat`/`exp` logic (30s safety margin, 10-minute expiry) - Properly forwards the `timeDifference` parameter for clock drift correction - Supports both PKCS#1 and PKCS#8 private key formats (via `node:crypto`) ### Changes - `lambdas/functions/control-plane/src/github/auth.ts` — replace `privateKey` with `createJwt` callback in `createAuth()` - `lambdas/functions/control-plane/src/github/auth.test.ts` — update tests to assert `createJwt` instead of `privateKey`, add test verifying unique JWTs with `jti` ### Test coverage - Existing tests updated to verify `createJwt` callback is passed instead of `privateKey` - New test generates two JWTs in rapid succession and verifies they differ (proving `jti` uniqueness) - New test validates JWT structure (header.payload.signature) and verifies `jti`, `iat`, `exp`, `iss` claims are present - All 343 control-plane tests pass Fixes github-aws-runners#5025
LudovicTOURMAN
pushed a commit
to doctolib-lab/terraform-aws-github-runner
that referenced
this pull request
Apr 7, 2026
🤖 I have created a release *beep* *boop* --- ## [7.4.1](github-aws-runners/terraform-aws-github-runner@v7.4.0...v7.4.1) (2026-03-09) ### Bug Fixes * gracefully handle JIT config failures and terminate unconfigured instance ([github-aws-runners#4990](github-aws-runners#4990)) ([c171550](github-aws-runners@c171550)) * **install-runner.sh:** support Debian ([github-aws-runners#5027](github-aws-runners#5027)) ([7755b7f](github-aws-runners@7755b7f)) * **lambda:** add jti claim to GitHub App JWTs to prevent concurrent collisions ([github-aws-runners#5056](github-aws-runners#5056)) ([07bd193](github-aws-runners@07bd193)), closes [github-aws-runners#5025](github-aws-runners#5025) * **lambda:** bump @octokit/auth-app from 8.1.2 to 8.2.0 in /lambdas in the octokit group ([github-aws-runners#5035](github-aws-runners#5035)) ([1c8083e](github-aws-runners@1c8083e)) * **lambda:** bump axios from 1.13.2 to 1.13.5 in /lambdas ([github-aws-runners#5028](github-aws-runners#5028)) ([0335e3a](github-aws-runners@0335e3a)) * **lambda:** bump qs from 6.14.1 to 6.14.2 in /lambdas ([github-aws-runners#5032](github-aws-runners#5032)) ([6dc97d5](github-aws-runners@6dc97d5)) * **lambda:** bump rollup from 4.46.2 to 4.59.0 in /lambdas ([github-aws-runners#5052](github-aws-runners#5052)) ([1e798b1](github-aws-runners@1e798b1)) * **lambda:** bump the aws group in /lambdas with 7 updates ([github-aws-runners#5021](github-aws-runners#5021)) ([c3c158d](github-aws-runners@c3c158d)) * **lambda:** bump the aws-powertools group in /lambdas with 4 updates ([github-aws-runners#5022](github-aws-runners#5022)) ([e8369cf](github-aws-runners@e8369cf)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: runners-releaser[bot] <194412594+runners-releaser[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes concurrent JWT collisions that cause silent job loss during burst workloads.
When multiple scale-up Lambda invocations generate GitHub App JWTs within the same second,
universal-github-app-jwtproduces byte-identical tokens (sameiat,exp,iss, nojti). GitHub rejects the duplicates, returning HTTP 404 onPOST /app/installations/{id}/access_tokens, which triggers silent batch dropping.Root cause
universal-github-app-jwtgenerates JWTs with only{ iat, exp, iss }claims. Theiatuses seconds precision (Math.floor(Date.now() / 1000)). With the same App ID and private key, concurrent invocations within the same second produce identical tokens.Fix
Replace
privateKey-based auth with a customcreateJwtcallback — a first-class API in@octokit/auth-appv8.x that completely bypassesuniversal-github-app-jwt. The callback:node:crypto.createSign(zero new dependencies)crypto.randomUUID()jticlaim, ensuring every token is uniqueiat/explogic (30s safety margin, 10-minute expiry)timeDifferenceparameter for clock drift correctionnode:crypto)Changes
lambdas/functions/control-plane/src/github/auth.ts— replaceprivateKeywithcreateJwtcallback increateAuth()lambdas/functions/control-plane/src/github/auth.test.ts— update tests to assertcreateJwtinstead ofprivateKey, add test verifying unique JWTs withjtiTest coverage
createJwtcallback is passed instead ofprivateKeyjtiuniqueness)jti,iat,exp,issclaims are presentFixes #5025