Skip to content

fix(lambda): add jti claim to GitHub App JWTs to prevent concurrent collisions#5056

Merged
Brend-Smits merged 1 commit into
github-aws-runners:mainfrom
closient:fix/jwt-collision-jti
Mar 6, 2026
Merged

fix(lambda): add jti claim to GitHub App JWTs to prevent concurrent collisions#5056
Brend-Smits merged 1 commit into
github-aws-runners:mainfrom
closient:fix/jwt-collision-jti

Conversation

@jensenbox
Copy link
Copy Markdown
Contributor

Summary

Fixes concurrent JWT collisions that cause silent job loss during burst workloads.

When multiple scale-up Lambda invocations generate GitHub App JWTs within the same second, universal-github-app-jwt produces byte-identical tokens (same iat, exp, iss, no jti). GitHub rejects the duplicates, returning HTTP 404 on POST /app/installations/{id}/access_tokens, which triggers silent batch dropping.

Root cause

universal-github-app-jwt generates JWTs with only { iat, exp, iss } claims. The iat uses seconds precision (Math.floor(Date.now() / 1000)). With the same App ID and private key, concurrent invocations within the same second produce identical tokens.

Fix

Replace privateKey-based auth with a custom createJwt callback — a first-class API in @octokit/auth-app v8.x that completely bypasses universal-github-app-jwt. The callback:

  • Signs JWTs using node:crypto.createSign (zero new dependencies)
  • Includes a crypto.randomUUID() jti claim, ensuring every token is unique
  • Preserves the existing iat/exp logic (30s safety margin, 10-minute expiry)
  • Properly forwards the timeDifference parameter for clock drift correction
  • Supports both PKCS#1 and PKCS#8 private key formats (via node:crypto)

Changes

  • lambdas/functions/control-plane/src/github/auth.ts — replace privateKey with createJwt callback in createAuth()
  • lambdas/functions/control-plane/src/github/auth.test.ts — update tests to assert createJwt instead of privateKey, add test verifying unique JWTs with jti

Test coverage

  • Existing tests updated to verify createJwt callback is passed instead of privateKey
  • New test generates two JWTs in rapid succession and verifies they differ (proving jti uniqueness)
  • New test validates JWT structure (header.payload.signature) and verifies jti, iat, exp, iss claims are present
  • All 343 control-plane tests pass

Fixes #5025

…ollisions

When multiple scale-up Lambda invocations run concurrently, the JWT
generation via universal-github-app-jwt produces byte-identical tokens
because it only includes iat, exp, and iss claims with second-precision
timestamps. GitHub rejects duplicate JWTs, returning HTTP 404 on the
installation token endpoint, which causes silent batch dropping.

Replace the privateKey-based auth with a custom createJwt callback using
the @octokit/auth-app first-class API. The callback signs JWTs with
node:crypto (zero new dependencies) and includes a randomUUID() jti
claim, ensuring every token is unique even within the same second.

Fixes github-aws-runners#5025
@jensenbox jensenbox requested a review from a team as a code owner March 6, 2026 08:12
Copy link
Copy Markdown
Contributor

@Brend-Smits Brend-Smits left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey!
Thanks for this great contribution 🚀
Tested this in our testing environment and things seem to work well. Okay from me (and @stuartp44) side.

@Brend-Smits Brend-Smits merged commit 07bd193 into github-aws-runners:main Mar 6, 2026
8 checks passed
npalm pushed a commit that referenced this pull request Mar 9, 2026
🤖 I have created a release *beep* *boop*
---


##
[7.4.1](v7.4.0...v7.4.1)
(2026-03-09)


### Bug Fixes

* gracefully handle JIT config failures and terminate unconfigured
instance
([#4990](#4990))
([c171550](c171550))
* **install-runner.sh:** support Debian
([#5027](#5027))
([7755b7f](7755b7f))
* **lambda:** add jti claim to GitHub App JWTs to prevent concurrent
collisions
([#5056](#5056))
([07bd193](07bd193)),
closes
[#5025](#5025)
* **lambda:** bump @octokit/auth-app from 8.1.2 to 8.2.0 in /lambdas in
the octokit group
([#5035](#5035))
([1c8083e](1c8083e))
* **lambda:** bump axios from 1.13.2 to 1.13.5 in /lambdas
([#5028](#5028))
([0335e3a](0335e3a))
* **lambda:** bump qs from 6.14.1 to 6.14.2 in /lambdas
([#5032](#5032))
([6dc97d5](6dc97d5))
* **lambda:** bump rollup from 4.46.2 to 4.59.0 in /lambdas
([#5052](#5052))
([1e798b1](1e798b1))
* **lambda:** bump the aws group in /lambdas with 7 updates
([#5021](#5021))
([c3c158d](c3c158d))
* **lambda:** bump the aws-powertools group in /lambdas with 4 updates
([#5022](#5022))
([e8369cf](e8369cf))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: runners-releaser[bot] <194412594+runners-releaser[bot]@users.noreply.github.com>
Brend-Smits pushed a commit that referenced this pull request Mar 11, 2026
🤖 I have created a release *beep* *boop*
---


##
[7.4.1](v7.4.0...v7.4.1)
(2026-03-09)


### Bug Fixes

* gracefully handle JIT config failures and terminate unconfigured
instance
([#4990](#4990))
([c171550](c171550))
* **install-runner.sh:** support Debian
([#5027](#5027))
([7755b7f](7755b7f))
* **lambda:** add jti claim to GitHub App JWTs to prevent concurrent
collisions
([#5056](#5056))
([07bd193](07bd193)),
closes
[#5025](#5025)
* **lambda:** bump @octokit/auth-app from 8.1.2 to 8.2.0 in /lambdas in
the octokit group
([#5035](#5035))
([1c8083e](1c8083e))
* **lambda:** bump axios from 1.13.2 to 1.13.5 in /lambdas
([#5028](#5028))
([0335e3a](0335e3a))
* **lambda:** bump qs from 6.14.1 to 6.14.2 in /lambdas
([#5032](#5032))
([6dc97d5](6dc97d5))
* **lambda:** bump rollup from 4.46.2 to 4.59.0 in /lambdas
([#5052](#5052))
([1e798b1](1e798b1))
* **lambda:** bump the aws group in /lambdas with 7 updates
([#5021](#5021))
([c3c158d](c3c158d))
* **lambda:** bump the aws-powertools group in /lambdas with 4 updates
([#5022](#5022))
([e8369cf](e8369cf))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: runners-releaser[bot] <194412594+runners-releaser[bot]@users.noreply.github.com>
Brend-Smits pushed a commit that referenced this pull request Apr 1, 2026
🤖 I have created a release *beep* *boop*
---


##
[7.4.1](v7.4.0...v7.4.1)
(2026-03-09)


### Bug Fixes

* gracefully handle JIT config failures and terminate unconfigured
instance
([#4990](#4990))
([c171550](c171550))
* **install-runner.sh:** support Debian
([#5027](#5027))
([7755b7f](7755b7f))
* **lambda:** add jti claim to GitHub App JWTs to prevent concurrent
collisions
([#5056](#5056))
([07bd193](07bd193)),
closes
[#5025](#5025)
* **lambda:** bump @octokit/auth-app from 8.1.2 to 8.2.0 in /lambdas in
the octokit group
([#5035](#5035))
([1c8083e](1c8083e))
* **lambda:** bump axios from 1.13.2 to 1.13.5 in /lambdas
([#5028](#5028))
([0335e3a](0335e3a))
* **lambda:** bump qs from 6.14.1 to 6.14.2 in /lambdas
([#5032](#5032))
([6dc97d5](6dc97d5))
* **lambda:** bump rollup from 4.46.2 to 4.59.0 in /lambdas
([#5052](#5052))
([1e798b1](1e798b1))
* **lambda:** bump the aws group in /lambdas with 7 updates
([#5021](#5021))
([c3c158d](c3c158d))
* **lambda:** bump the aws-powertools group in /lambdas with 4 updates
([#5022](#5022))
([e8369cf](e8369cf))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: runners-releaser[bot] <194412594+runners-releaser[bot]@users.noreply.github.com>
LudovicTOURMAN pushed a commit to doctolib-lab/terraform-aws-github-runner that referenced this pull request Apr 7, 2026
…ollisions (github-aws-runners#5056)

## Summary

Fixes concurrent JWT collisions that cause silent job loss during burst
workloads.

When multiple scale-up Lambda invocations generate GitHub App JWTs
within the same second, `universal-github-app-jwt` produces
byte-identical tokens (same `iat`, `exp`, `iss`, no `jti`). GitHub
rejects the duplicates, returning HTTP 404 on `POST
/app/installations/{id}/access_tokens`, which triggers silent batch
dropping.

### Root cause

`universal-github-app-jwt` generates JWTs with only `{ iat, exp, iss }`
claims. The `iat` uses seconds precision (`Math.floor(Date.now() /
1000)`). With the same App ID and private key, concurrent invocations
within the same second produce identical tokens.

### Fix

Replace `privateKey`-based auth with a custom `createJwt` callback — a
first-class API in `@octokit/auth-app` v8.x that completely bypasses
`universal-github-app-jwt`. The callback:

- Signs JWTs using `node:crypto.createSign` (zero new dependencies)
- Includes a `crypto.randomUUID()` `jti` claim, ensuring every token is
unique
- Preserves the existing `iat`/`exp` logic (30s safety margin, 10-minute
expiry)
- Properly forwards the `timeDifference` parameter for clock drift
correction
- Supports both PKCS#1 and PKCS#8 private key formats (via
`node:crypto`)

### Changes

- `lambdas/functions/control-plane/src/github/auth.ts` — replace
`privateKey` with `createJwt` callback in `createAuth()`
- `lambdas/functions/control-plane/src/github/auth.test.ts` — update
tests to assert `createJwt` instead of `privateKey`, add test verifying
unique JWTs with `jti`

### Test coverage

- Existing tests updated to verify `createJwt` callback is passed
instead of `privateKey`
- New test generates two JWTs in rapid succession and verifies they
differ (proving `jti` uniqueness)
- New test validates JWT structure (header.payload.signature) and
verifies `jti`, `iat`, `exp`, `iss` claims are present
- All 343 control-plane tests pass

Fixes github-aws-runners#5025
LudovicTOURMAN pushed a commit to doctolib-lab/terraform-aws-github-runner that referenced this pull request Apr 7, 2026
🤖 I have created a release *beep* *boop*
---


##
[7.4.1](github-aws-runners/terraform-aws-github-runner@v7.4.0...v7.4.1)
(2026-03-09)


### Bug Fixes

* gracefully handle JIT config failures and terminate unconfigured
instance
([github-aws-runners#4990](github-aws-runners#4990))
([c171550](github-aws-runners@c171550))
* **install-runner.sh:** support Debian
([github-aws-runners#5027](github-aws-runners#5027))
([7755b7f](github-aws-runners@7755b7f))
* **lambda:** add jti claim to GitHub App JWTs to prevent concurrent
collisions
([github-aws-runners#5056](github-aws-runners#5056))
([07bd193](github-aws-runners@07bd193)),
closes
[github-aws-runners#5025](github-aws-runners#5025)
* **lambda:** bump @octokit/auth-app from 8.1.2 to 8.2.0 in /lambdas in
the octokit group
([github-aws-runners#5035](github-aws-runners#5035))
([1c8083e](github-aws-runners@1c8083e))
* **lambda:** bump axios from 1.13.2 to 1.13.5 in /lambdas
([github-aws-runners#5028](github-aws-runners#5028))
([0335e3a](github-aws-runners@0335e3a))
* **lambda:** bump qs from 6.14.1 to 6.14.2 in /lambdas
([github-aws-runners#5032](github-aws-runners#5032))
([6dc97d5](github-aws-runners@6dc97d5))
* **lambda:** bump rollup from 4.46.2 to 4.59.0 in /lambdas
([github-aws-runners#5052](github-aws-runners#5052))
([1e798b1](github-aws-runners@1e798b1))
* **lambda:** bump the aws group in /lambdas with 7 updates
([github-aws-runners#5021](github-aws-runners#5021))
([c3c158d](github-aws-runners@c3c158d))
* **lambda:** bump the aws-powertools group in /lambdas with 4 updates
([github-aws-runners#5022](github-aws-runners#5022))
([e8369cf](github-aws-runners@e8369cf))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: runners-releaser[bot] <194412594+runners-releaser[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Concurrent Lambda invocations generate byte-identical JWTs causing transient 404 on installation token endpoint

2 participants