Description
Motivation.
Summary
Exploring tool_calls in VLLM. Seems to be a need for additional handling of non-iterable None type responses in the tool_calls field.
Motivation
I've been testing with Google's new ADK (backed by LiteLLM) against a VLLM hosted Qwen 2.5 Instruct model and the Hermes parser. (Using Kserve).
https://github.com/google/adk-python/blob/290058eb05211ef531b1752c6290da3f365e4e73/src/google/adk/models/lite_llm.py#L194
ADK explicitly returns a None in the tool_calls field.
tool_calls=tool_calls or None,
From what I understand, some models and hosting do expect this None type, however VLLM does not. This means that on the second message of the chat, you get the following:
ERROR - fast_api.py:616 - Error in event_generator: litellm.BadRequestError: OpenAIException - 'NoneType' object is not iterable
This seems to bring us to
vllm/vllm/entrypoints/chat_utils.py
Line 1072 in 54a66e5
in which the iteration takes place here.
vllm/vllm/entrypoints/chat_utils.py
Line 1097 in 54a66e5
It would be nice if VLLM could work cleanly with ADK and other tool calling solutions that make use of the None type. The tool calls themselves seem to work well, this is an issue of follow up chats after no tool calls.
Proposed Change.
Currently I am doing a conditional modification to the ADK LiteLLM, modifying the None to [] for models where appropriate, but it seems like it would be cleaner if VLLM could handle a None type in the tool_calls field.
vllm/vllm/entrypoints/chat_utils.py
Line 1097 in 54a66e5
Becomes ->
if "tool_calls" in parsed_msg and parsed_msg["tool_calls"] is not None:
I am fairly new to VLLM and want to make sure I'm not missing anything before a pull request goes in. Are there any larger implications of this change that anyone can see? If anyone has a more elegant solution that would also be appreciated.
Feedback Period.
A week seems fine. Sanity check/alternate solution methods appreciated.
CC List.
No response
Any Other Things.
No response
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.