Skip to content

feat(model): add cached_tokens support to ChatUsage for all providers#1343

Open
YingchaoX wants to merge 1 commit intoagentscope-ai:mainfrom
YingchaoX:feature/chat-usage-cached-tokens
Open

feat(model): add cached_tokens support to ChatUsage for all providers#1343
YingchaoX wants to merge 1 commit intoagentscope-ai:mainfrom
YingchaoX:feature/chat-usage-cached-tokens

Conversation

@YingchaoX
Copy link
Copy Markdown
Contributor

AgentScope Version

1.0.17

Description

Background

Issue #1324 requested support for cached_tokens in ChatUsage to track prompt caching metrics from
OpenAI SDK's prompt_tokens_details.cached_tokens. This is important for monitoring cache hit rates a
nd optimizing costs.

Changes Made

  • Add cached_tokens field to ChatUsage dataclass
  • Implement cached_tokens extraction for OpenAI, Anthropic, Gemini, DashScope
  • Only use cache_read_input_tokens for Anthropic (not cache_creation) to avoid false positives on
    cold requests
  • Preserve 0 values from APIs (don't convert to None)
  • Support both dict and object formats in API responses
  • Add tracing attribute AGENTSCOPE_USAGE_CACHED_TOKENS
  • Update tracing extractor to report cached_tokens
  • Add comprehensive tests for all providers

How to Test

import asyncio
from agentscope.model import OpenAIChatModel, DashScopeChatModel

async def test():
    # OpenAI example
    model = OpenAIChatModel(
        model_name="gpt-4",
        api_key="your-key",
    )
    response = await model([{"role": "user", "content": "Hello"}])
    print(f"cached_tokens: {response.usage.cached_tokens}")

    # DashScope example
    model = DashScopeChatModel(
        model_name="qwen-turbo",
        api_key="your-key",
    )
    response = await model([{"role": "user", "content": "Hello"}])
    print(f"cached_tokens: {response.usage.cached_tokens}")

asyncio.run(test())

Testing Status

• ✅ OpenAI: Unit tested
• ✅ DashScope: Tested with actual API
• ✅ Anthropic: Tested via DashScope Anthropic-compatible endpoint
⚠️ Gemini: Code implementation complete (needs community testing with actual API)

Checklist

• [x] Code has been formatted with pre-commit run --all-files command
• [x] All tests are passing
• [x] Docstrings are in Google style
• [ ] Related documentation has been updated (e.g. links, examples, etc.)
• [x] Code is ready for review

Notes

The Gemini implementation follows the same pattern as other providers and has been code-reviewed, but
could benefit from testing by someone with Gemini API access.

Fixes #1324

- Add cached_tokens field to ChatUsage dataclass
- Implement cached_tokens extraction for OpenAI, Anthropic, Gemini, DashScope
- Only use cache_read_input_tokens for Anthropic (not cache_creation)
- Preserve 0 values from APIs (don't convert to None)
- Support both dict and object formats in API responses
- Add tracing attribute AGENTSCOPE_USAGE_CACHED_TOKENS
- Update tracing extractor to report cached_tokens
- Add comprehensive tests for all providers

Fixes agentscope-ai#1324
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: ChatUsage 支持 cached_token

1 participant