Skip to content

fix(sdk/python): report a provider-confirmed zero cost instead of unknown - #1105

Merged
ZeroPoint95 merged 1 commit into
Agent-Field:mainfrom
remote-controlled-man:codex/agentfield-provider-zero-cost
Oct 17, 2026
Merged

ZeroPoint95 merged 1 commit into
Agent-Field:mainfrom
remote-controlled-man:codex/agentfield-provider-zero-cost

Conversation

@remote-controlled-man

Copy link
Copy Markdown
Contributor

Summary

_resolve_cost() prefers a provider-native cost figure — OpenRouter's usage.cost, which this SDK asks for on every OpenRouter request (openrouter_attribution.apply_openrouter_usage_accounting sets usage: {"include": true}) — but it accepted that figure only when it was strictly positive:

# 1. Provider-native cost (e.g. OpenRouter returns usage.cost in USD/credits).
if native is not None:
    cost = float(native)
    if cost > 0:          # a provider-reported 0.0 is discarded here
        return cost, "provider"

So a provider that reports the call's cost as exactly 0 — the zero-priced / free-tier case — falls through to the two LiteLLM branches, and when those have no price either, _resolve_cost returns (None, None). detect_multimodal_response() then hands the caller cost_usd=None, cost_source=None; agent_ai.py records that into the execution's CostTracker; and CostTracker.serialize() leaves any_cost=False and emits total_cost_usd: null. The cost/usage surface reports "cost unknown" for a call whose cost the provider confirmed as $0 — including a run whose calls were all free, which is exactly the run where a user most wants to see 0.0 rather than a hole.

This PR accepts cost >= 0 in the provider-native branch only. The two LiteLLM branches keep > 0, which is still correct there: litellm.completion_cost() returns 0.0 for models absent from its pricing database, so a zero from LiteLLM means "unpriced" while a zero from the provider means "free".

Why this is a bug rather than the intended contract

Three places in this repository already treat a known zero as a value:

  • _resolve_cost's own docstring reserves (None, None) for "when no cost can be determined". A provider-reported $0 is determined.
  • CostTracker.serialize() is explicitly built to express a known zero — if e.cost_usd is not None: total_cost += e.cost_usd; any_cost = True → total_cost_usd: 0.0 — and the recording site in agent_ai.py documents cost_usd=None as "when the price is unknown".
  • The sibling recording paths use presence, not positivity, for a provider figure: agent.py records cost_source="provider" if cost is not None else None, and media_providers.py does the same for its metadata figure.

The TypeScript SDK already accepts a real zero for the same provider figure: sdk/typescript/src/usage/aiUsage.ts costFromUsageRaw() returns it when cost >= 0, and recordAiSdkUsage's docstring states "Known zero receipts remain reported entries". So today the two SDKs disagree about the same call — TS reports 0, Python reports null.

Type of change

  • Bug fix
  • New feature
  • Refactor / cleanup
  • Docs only
  • Tests only
  • CI / tooling
  • Breaking change

Test plan

Two new regression tests, both failing on the unmodified base with assert None == 0.0:

  • cd sdk/python && python -m pytest tests/test_usage_transport.py --no-cov → 45 passed. New cases: TestCostResolution::test_openrouter_native_zero_cost_is_known (the resolver returns (0.0, "provider")) and TestDetectMultimodalUsage::test_openrouter_native_zero_cost_read_through (the public detect_multimodal_response() reports cost_usd == 0.0 and cost_source == "provider" with tokens intact). Both keep LiteLLM mocked as raising, so a pass can only come from the provider branch.
  • Cost/usage/multimodal surface, 14 test files (test_usage_transport, test_cost_tracker, test_multimodal_response, test_multimodal_response_additional, test_multimodal_response_cost, test_data_url_media_outputs, test_media_integration, test_media_providers, test_openrouter_audio, test_openrouter_video, test_usage_stream, test_video_output, test_vision, test_issue_589) → 366 passed.
  • ruff check . → All checks passed! with the CI-pinned ruff==0.15.22; mypy --config-file mypy.ini agentfield/ → Success: no issues found in 83 source files with mypy==1.18.2 from the dev extra. Python 3.12.7 on Windows.
  • Full Python SDK suite, unfiltered except the hanging module (measured on this identical patch immediately before its rebase onto 9bd0a0dd; git range-diff reports = and the stable patch-id is unchanged): 12 failed, 2506 passed, 6 skipped, 38 deselected. All 12 are pre-existing host artifacts, attributed by node ID rather than by name: re-running the same nodes with this patch stashed (target blob back to 1ac98a55) produced the identical 10 deterministic failures — 7 × test_log_writer (CRLF / pipe-timing), 2 × test_harness_packaging (GBK UnicodeDecodeError from pathlib.read_text), 1 × test_tool_calling.py::TestExecuteToolCallLoop::test_single_tool_call_then_response — and the remaining 2 (test_litellm_observability) pass on warm re-runs on both trees, which is that file's known cold-cache flake. Recorded run at this head, with those twelve nodes deselected: 2506 passed, 6 skipped, 50 deselected (the 50 are the 38 marker-deselected tests plus those twelve), i.e. zero failures once the proven host artifacts are set aside.
  • tests/test_agent_mesh.py is excluded because it does not complete on this Windows host — a pre-existing hang, unrelated to this change. Left to CI on ubuntu-latest.
  • ./scripts/test-all.sh and ./scripts/coverage-summary.sh were not run locally: they also build the Go control plane, the web UI and the desktop app, and the coverage summary needs a Docker engine, which is unavailable here. This PR changes only sdk/python/**, and .github/workflows/sdk-python.yml filters on exactly that path plus its own workflow file, so the Python surface above is the applicable CI.
  • ruff format was deliberately not applied. On pristine main, ruff format --diff already proposes two unrelated hunks in exactly these two files — _resolve_cost's signature line and a HarnessResult(...) call in the tests — so reformatting would add pre-existing churn to a one-line fix. CI's format step is continue-on-error: true.

Test coverage

  • I ran tests for the surface(s) I changed locally.
  • New code paths are covered by tests in this PR (no bare additions).
  • If I removed code, I updated coverage-baseline.json — not applicable; nothing was removed and no source line lost coverage.
  • The coverage gate check is green in CI before requesting review — left to CI, because ./scripts/coverage-summary.sh measures all five surfaces and needs a Docker engine.

Heads-up so the gate output is not misread: agentfield.multimodal_response is not in the --cov whitelist in sdk/python/pyproject.toml's pytest addopts (which lists nine modules), and scripts/coverage-surface.sh measures sdk-python by running bare pytest, so this file never reaches the coverage report that patch-coverage-gate.sh diffs against. This PR will therefore report sdk-python | 0 | — | ➖ no changes. That is the whitelist, not a missing test: the changed branch is exercised directly through _resolve_cost and through the public detect_multimodal_response() entry point, and the neighbouring positive provider-cost cases (0.0042, 0.0012) were already pinned. pyproject.toml and coverage-baseline.json are left untouched to keep this to one seam — adding --cov=agentfield.multimodal_response to addopts would let the gate see this module, and I can send that separately if you want it.

Checklist

  • I have read CONTRIBUTING.md and docs/DEVELOPMENT.md.
  • Commits are signed and follow conventional-commits style. — the subject follows conventional-commits; the commit is not signed, as I have no signing key configured. Flagging it rather than ticking the box.
  • I have linked any related issues — there is none. I found this by reading the cost-resolution path, then searched open and closed issues and PRs for _resolve_cost, cost_source, multimodal_response, cost_usd provider, zero cost and free model cost: no report and no competing change. Happy to file one first if you would rather track it that way.

Related issues / PRs

None. No file overlap with any open PR: the Python files under review elsewhere are agent.py/agent_server.py/logger.py (#1090), the prompt-template surface (#1092) and the harness CLI (#1104); neither sdk/python/agentfield/multimodal_response.py nor sdk/python/tests/test_usage_transport.py is touched by any of them.

…nown

_resolve_cost() accepted a provider-native cost figure only when it was
strictly positive, so a provider-reported 0.0 -- what a zero-priced or
free-tier model returns -- was discarded and resolution fell through to the
LiteLLM branches, ending at (None, None). detect_multimodal_response() then
reported cost_usd=None/cost_source=None, agent_ai.py recorded that into the
execution CostTracker, and serialize() emitted total_cost_usd: null for a run
whose calls were all free.

Accept cost >= 0 in the provider-native branch only. The two LiteLLM branches
keep > 0 because litellm.completion_cost() returns 0.0 for models absent from
its pricing database, so a zero there means "unpriced" rather than "free".
This also matches the TypeScript SDK, which accepts a real zero for the same
provider figure (costFromUsageRaw: cost >= 0, "known zero receipts remain
reported entries").
Copilot AI balanced review requested due to automatic review settings October 15, 2026 16:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@github-actions

Copy link
Copy Markdown
Contributor

Performance

SDK Memory Δ Latency Δ Tests Status
Python 9.1 KB - 0.17 µs -51% ✓ ✓

✓ No regressions detected

@github-actions

Copy link
Copy Markdown
Contributor

📊 Coverage gate

Thresholds from .coverage-gate.toml: per-surface ≥ 84%, aggregate ≥ 85%, max per-surface regression ≤ 1.0 pp, max aggregate regression ≤ 0.50 pp.

Surface Current Baseline Δ
control-plane 88.00% 87.40% ↑ +0.60 pp 🟡
sdk-go 93.30% 92.00% ↑ +1.30 pp 🟢
sdk-python 94.72% 93.73% ↑ +0.99 pp 🟢
sdk-typescript 91.77% 90.42% ↑ +1.35 pp 🟢
web-ui 84.77% 84.79% ↓ -0.02 pp 🟡
aggregate 85.96% 85.75% ↑ +0.21 pp 🟡

✅ Gate passed

No surface regressed past the allowed threshold and the aggregate stayed above the floor.

@github-actions

Copy link
Copy Markdown
Contributor

📐 Patch coverage gate

Threshold: 80% on lines this PR touches vs origin/main (from .coverage-gate.toml:thresholds.min_patch).

Surface Touched lines Patch coverage Status
control-plane 0 — ➖ no changes
sdk-go 0 — ➖ no changes
sdk-python 0 — ➖ no changes
sdk-typescript 0 — ➖ no changes
web-ui 0 — ➖ no changes

✅ Patch gate passed

Every surface whose lines were touched by this PR has patch coverage at or above the threshold.

@ZeroPoint95 ZeroPoint95 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@remote-controlled-man, thanks for the focused fix. Provider-reported zero now stays a known cost through the public response and serialized usage receipt, while the unknown-pricing fallback remains intact. The regression checks reproduce the bug on the base and pass here. Looks good from the code-review side.

@ZeroPoint95
ZeroPoint95 merged commit 07e148d into Agent-Field:main Oct 17, 2026
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants