feat: add compact API support for OpenAI#34926
feat: add compact API support for OpenAI#34926constance (jiaming2li) wants to merge 6 commits intolangchain-ai:masterfrom
Conversation
- Add tokens_per_image parameter for fixed image token penalty - Handle list-based content blocks (text, image, image_url) - Prevent massive overestimation from base64-encoded images - Maintain backward compatibility with string content - Add comprehensive test coverage for multimodal scenarios Fixes token counting for multimodal messages where base64 images were previously counted as 25,000+ tokens instead of ~85 tokens.
- Add tokens_per_image parameter for fixed image token penalty - Handle list-based content blocks (text, image, image_url) - Prevent massive overestimation from base64-encoded images - Maintain backward compatibility with string content - Add comprehensive test coverage for multimodal scenarios
constance (jiaming2li)
requested review from
ccurme (ccurme),
Eugene Yurtsev (eyurtsev) and
Mason Daugherty (mdrxy)
as code owners
github-actions
bot
added
external
integration
Merging this PR will not alter performance
Comparing 
Footnotes
|
|
constance (@jiaming2li) would you mind resolving all CI failures and conflicts? |
I‘ll fix them tomorrow. |
github-actions
bot
added
langchain
constance (jiaming2li)
changed the title
github-actions
bot
added
the
feature
|
Hello, this is great! any updates on what it takes to get this merged? |
Hi! Thanks for the follow-up. I have just updated the branch with the latest master, and it is now ready to be merged. |
ccurme (ccurme)
left a comment
ccurme (ccurme)
left a comment
There was a problem hiding this comment.
Hello, thanks very much for this.
ChatOpenAI now supports automatic server-side compaction via #35212.
There may still be reason to support "manual" compaction (direct use through the compact endpoints), but we'd need a design doc for this. The implementation here has a number of problems:
- We need to instantiate a separate
ChatOpenAI(use_compact_api=True)to use it. It may make more sense as acompactmethod. - We return an artificial AIMessage with content
"[Conversation compacted]". There's no reason to add this content to the message history. We need end-to-end usage examples showing how to usemodel.compact(or similar) in a multi-turn conversation.
If #35212 is insufficient and we do need support for manual compaction, would you mind opening an issue explaining the use case and proposing a design?
4 participants
Summary: This PR introduces support for OpenAI's new /responses/compact endpoint. This allows for efficient context management by compacting message history directly through the API.
close #34924
Changes:
Added use_compact_api and compact_target_tokens parameters to ChatOpenAI.
Implemented routing logic in _generate and _agenerate to use the compact endpoint when requested.
Added _construct_compact_result to parse the specific response format of the compact API.