Skip to content

Credential pools: no supported way to rotate a healthy credential before its quota window closes (Codex) #71095

Description

@xMikeMickelson

Problem

Credential-pool rotation is reactive only: an entry leaves rotation when a request fails with a quota/auth signature (CredentialPool.mark_exhausted_and_rotate). There is no supported way to rotate before a window closes.

For an openai-codex pool on fill_first, that means a fleet necessarily spends the last few percent of its weekly window hitting the wall, and the in-flight sessions at that moment are the ones that die — even when a sibling credential on a different ChatGPT account is sitting at 0% used.

Measured on a two-account Pro pool:

credential account used window reset
A (priority 0, active) ab2774… 93% Jul 31 13:07Z
B (priority 1) 3320e7… 0% Jul 31 19:02Z

A full untouched week sat idle while the active credential ran to exhaustion.

Why the existing API can't express this

mark_exhausted_and_rotate cannot be used to pre-empt the wall, and correctly so. CredentialPool._codex_quota_restored_upstream live-probes the usage endpoint for any entry frozen with a 429/quota-shaped error and lifts the freeze when the account is actually usable — that is the fix for #43747, so a stale weekly last_error_reset_at cannot strand a recovered account.

A pre-emptive freeze at 93% is exactly that shape on an account that genuinely is usable, so the probe overturns it. Verified on v0.19.0: after mark_exhausted_and_rotate(status_code=429, error_context={"reset_at": <real window reset>}), two consecutive fresh processes re-selected the drained credential with last_status flipped back to ok.

So the two behaviours are in direct tension: "exhausted" is defined as provably unusable, and there is no separate way to say usable but deliberately not preferred.

What Hermes already has

agent/account_usage.py::_fetch_codex_account_usage returns used_percent and reset_at per window and accepts an explicit api_key, so it can already be asked about any pool entry, not just the active one. The data needed to make this decision is available; only the actuation is missing.

Proposed

A supported way to deprioritise a healthy credential, e.g. either:

  1. CredentialPool.demote(entry_id) / set_priority(entry_id, priority) — public reordering (the pool already performs exactly this reordering internally for STRATEGY_ROUND_ROBIN in _select_unlocked); or
  2. an opt-in credential_pool_quota_headroom policy: when a Codex entry's used_percent crosses a configurable threshold and a sibling has more headroom, prefer the sibling until the window resets.

Either shape lets operators drain accounts in a controlled order and keeps prompt-cache breaks to one per switch, instead of taking an uncontrolled break at the wall plus the loss of every in-flight session.

Workaround in use

Demoting the drained entry in the pool's priority order (mirroring the STRATEGY_ROUND_ROBIN reordering) — this is not probed away and holds across processes. Working, but it reaches into pool internals, which is why a supported API is worth having.

Environment

Hermes Agent v0.19.0 (2026.7.20), upstream a61183b5, Docker install, credential_pool_strategies: {openai-codex: fill_first}, two OAuth device-code grants on separate ChatGPT Pro accounts.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium — degraded but workaround existsarea/authAuthentication, OAuth, credential poolsarea/billingAccount usage, credit usage, billing (cross-cutting)comp/agentCore agent runtime: loop, agent_init, prompt builder, context-compression, responses endpointneeds-decisionAwaiting maintainer decision before any implementationprovider/openaiOpenAI / Codex Responses APItype/featureNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions