add function call parser for DeepSeek V3

[finger92]

pls look at the old pr #5054 for basic information

update on Apr 10

I have updated the code and have fixed the "Token id 129279" problem. but this is still not working with xgramma. The output seems to be unstable when using xgrammar, exp the output may be
"<｜tool▁calls▁begin｜><<<<<<<<<<<<<<<<<<<<<<"
and then the server crashed with this exception:

File "/sgl-workspace/sglang/python/sglang/srt/constrained/xgrammar_backend.py", line 59, in accept_token
    assert self.matcher.accept_token(token)

I also add a new chat template which inspired by "function calling" part in "https://huggingface.co/deepseek-ai/DeepSeek-V2.5"

deepseek2.jinja

{% if not add_generation_prompt is defined %}
    {% set add_generation_prompt = false %}
{% endif %}
{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='', is_first_sp=true, is_last_user=false) %}
{%- for message in messages %}
    {%- if message['role'] == 'system' %}
        {%- if ns.is_first_sp %}
            {% set ns.system_prompt = ns.system_prompt + message['content'] %}
            {% set ns.is_first_sp = false %}
        {%- else %}
            {% set ns.system_prompt = ns.system_prompt + '\n\n' + message['content'] %}
        {%- endif %}
    {%- endif %}
{%- endfor %}
{{ bos_token }}
{{ ns.system_prompt }}
{%- for message in messages %}
    {%- if message['role'] == 'user' %}
        {%- set ns.is_tool = false -%}
        {%- set ns.is_first = false -%}
        {%- set ns.is_last_user = true -%}
        {{'<｜User｜>' + message['content'] + '<｜Assistant｜>'}}
    {%- endif %}
    {%- if message['role'] == 'assistant' and tools is defined and tools is not none %}
        {%- set ns.is_last_user = false -%}
        {%- if ns.is_tool %}
            {{'<｜tool▁outputs▁end｜>'}}
        {%- endif %}
        {%- set ns.is_first = false %}
        {%- set ns.is_tool = false -%}
        {%- set ns.is_output_first = true %}
        {%- if message['content'] is not none %}
            {{message['content']}} 
        {%- endif %} 
        {{'## Tools\n\n### Function\n\nYou have the following functions available:\n\n'}}
        {%- for tool in tools %}
            {{'- `' + tool['function']['name'] + '`:\n```json'}} {{- tool['function'] | tojson }} {{'\n```'}}
            {%- set ns.is_first = true -%}
        {%- endfor %}
    {%- endif %}
    {%- if message['role'] == 'assistant' and (tools is not defined or tools is none)%}
        {%- set ns.is_last_user = false -%}
        {%- if ns.is_tool %}
            {{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}
            {%- set ns.is_tool = false -%}
        {%- else %}
            {% set content = message['content'] %}
            {{content + '<｜end▁of▁sentence｜>'}}
        {%- endif %}
    {%- endif %}
    {%- if message['role'] == 'tool' %}
        {%- set ns.is_last_user = false -%}
        {%- set ns.is_tool = true -%}
        {%- if ns.is_output_first %}
            {{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}
            {%- set ns.is_output_first = false %}
        {%- else %}
            {{'\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}
        {%- endif %}
    {%- endif %}
{%- endfor -%}
{% if ns.is_tool %}
    {{'<｜tool▁outputs▁end｜>'}}
{% endif %}
{% if add_generation_prompt and not ns.is_last_user and not ns.is_tool %}
    {{'<｜Assistant｜>'}}
{% endif %}

I did an function call test on these two templates and the result is

template	rate of success
deepseek.jinja	45%
deepseek2.jinja	100%

here is the test code

import requests, json, random, re

test_func = {
    "type": "function",
    "function": {
        "name": "query_weather",
        "description": "Get weather of an location, the user shoud supply a location first",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {
                    "type": "string",
                    "description": "The city, e.g. Beijing"
                }
            },
            "required": [
                "city"
            ]
        }
    }
}


req_base_url = "http://127.0.0.1:30000"
cities = [
    "Beijing", 
    "Chongqing",
    "Chengdu",
    "Dalian",
    "Guangzhou",
    "Hangzhou",
    "Harbin",
    "Hefei",
    "Kunming",
    "Lanzhou",
    "Nanjing",
    "Qingdao",
    "Shanghai",
    "Shenzhen",
    "Suzhou",
    "Tianjin",
    "Wuhan",
    "Xi'an",
    "Xiamen",
    "Zhengzhou"
]
user_content_tp = "Hows the weather like in {} today"
req_body = {
    "messages": [
        {
            "role": "user",
            "content": ""
        }
    ],
    "temperature": 0,
    "max_tokens": 100,
    "model": "deepseek-ai/DeepSeek-V3-0324",
    "tools": [
        test_func
    ]
}

total_failed = 0
for i in range(100):
    req_body["messages"][1]["content"] = user_content_tp.format(random.choice(cities))
    res = requests.post(req_base_url + "/v1/chat/completions", json=req_body)
    res = res.json()
    if len(res["choices"]) > 0:
        if len(res["choices"][0]["message"]["tool_calls"]) > 0:
            print("function call successfull: " + json.dumps(res["choices"][0]["message"]["tool_calls"][0]))
        else:
            total_failed += 1
            print("function call failed: " + json.dumps(res))
            
    # flush cache
    requests.get(req_base_url + "/flush_cache")
print(f"total_failed: {total_failed}")

[minleminzui]

could you add the document about tool calling of dspk v3， thanks. https://docs.sglang.ai/references/deepseek.html

[finger92]

could you add the document about tool calling of dspk v3， thanks. https://docs.sglang.ai/references/deepseek.html

Sure, I will add it today

[feng397]

Does it support streaming output？

[finger92]

Does it support streaming output？

No. Is there any scenario where stream needs to be used during function call?

[feng397]

Does it support streaming output？

No. Is there any scenario where stream needs to be used during function call?

I find it is supported in the official API:

ChatCompletionChunk(id='', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, refusal=None, role=None, tool_calls=[ChoiceDeltaToolCall(index=0, id='', function=ChoiceDeltaToolCallFunction(arguments='', name='get_weather'), type='function')]), finish_reason=None, index=0, logprobs=None)], created=, model='deepseek-chat', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_3d5141a69a_prod0225', usage=None)

ChatCompletionChunk(id='', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, refusal=None, role=None, tool_calls=[ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='{"', name=None), type=None)]), finish_reason=None, index=0, logprobs=None)], created=, model='deepseek-chat', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_3d5141a69a_prod0225', usage=None)

ChatCompletionChunk(id='', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, refusal=None, role=None, tool_calls=[ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='location', name=None), type=None)]), finish_reason=None, index=0, logprobs=None)], created=, model='deepseek-chat', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_3d5141a69a_prod0225', usage=None)

[moqimoqidea]

Does it support streaming output？

No. Is there any scenario where stream needs to be used during function call?

For example, Anthropic's Streaming request with tool use consists of a streaming message consisting of a text content (which may be empty) and a tool use part, and uses partial_json to combine the full functions and parameters of the tool use in multiple messages.

Thanks for your work, I think it would be awesome if sglang + deepseek v3 function call could support streaming.

[finger92]

could you add the document about tool calling of dspk v3， thanks. https://docs.sglang.ai/references/deepseek.html

I have added the doc.
I also add the chat template mentioned in this pr.

[finger92]

@feng397 @moqimoqidea Thanks for pointing out. I will check these. Maybe I can create another PR for implement "function calling with streaming request". Probably next week.

[minleminzui]

@finger92 is the chat_template of dpsk v3 the same with the one of dpsk v2.5? why did you use the chat_template of dpsk v2.5?

[minleminzui]

@feng397 @moqimoqidea Thanks for pointing out. I will check these. Maybe I can create another PR for implement "function calling with streaming request". Probably next week.

@feng397 @moqimoqidea @finger92 Streaming request with tool use has been realized in Sglang. No additional implementation is required for dpsk v3

[finger92]

@finger92 is chat_template of dpsk v3 the same the one of dpsk v2.5? why did you use the chat_template of dpsk v2.5?

Oh, I just check and see that they are different. I will update it to V3's template

[finger92]

@feng397 @moqimoqidea Thanks for pointing out. I will check these. Maybe I can create another PR for implement "function calling with streaming request". Probably next week.

@feng397 @moqimoqidea @finger92 Streaming request with tool use has been realized in Sglang. No additional implementation is required for dpsk v3

I see that. But the "function part" for deepseek output is not completely json format, so it has some compatible issues

[minleminzui]

@feng397 @moqimoqidea Thanks for pointing out. I will check these. Maybe I can create another PR for implement "function calling with streaming request". Probably next week.

@feng397 @moqimoqidea @finger92 Streaming request with tool use has been realized in Sglang. No additional implementation is required for dpsk v3

I see that. But the "function part" for deepseek output is not completely json format, so it has some compatible issues

ok，thanks， I'll check it， but please remove the deepseek.jinja in docs/references/deepseek.jinja, replace the url https://github.com/sgl-project/sglang/tree/main/docs/references/deepseek.jinja with https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/main/tokenizer_config.json#L34

[finger92]

@feng397 @moqimoqidea Thanks for pointing out. I will check these. Maybe I can create another PR for implement "function calling with streaming request". Probably next week.

@feng397 @moqimoqidea @finger92 Streaming request with tool use has been realized in Sglang. No additional implementation is required for dpsk v3

I see that. But the "function part" for deepseek output is not completely json format, so it has some compatible issues

ok，thanks， I'll check it， but please remove the deepseek.jinja in docs/references/deepseek.jinja, replace the url https://github.com/sgl-project/sglang/tree/main/docs/references/deepseek.jinja with https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/main/tokenizer_config.json#L34

The chat template I submitted is an adapted version of V3's.
Using the original chat template, my idea is to add a function call prompt to the message in adapter.py when the tools appear in request. Do you think this is a good idea?

[minleminzui]

@feng397 @moqimoqidea Thanks for pointing out. I will check these. Maybe I can create another PR for implement "function calling with streaming request". Probably next week.

@feng397 @moqimoqidea @finger92 Streaming request with tool use has been realized in Sglang. No additional implementation is required for dpsk v3

I see that. But the "function part" for deepseek output is not completely json format, so it has some compatible issues

ok，thanks， I'll check it， but please remove the deepseek.jinja in docs/references/deepseek.jinja, replace the url https://github.com/sgl-project/sglang/tree/main/docs/references/deepseek.jinja with https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/blob/main/tokenizer_config.json#L34

The chat template I submitted is an adapted version of V3's. Using the original chat template, my idea is to add a function call prompt to the message in adapter.py when the tools appear in request. Do you think this is a good idea?

@finger92 I think it is better not to change its original chat template, please use the original chat template, change the code，thanks

[finger92]

@zhaochenyang20 Hi, thanks for trigging CI. I think the failed check is not related to this PR. what do you think?

[zhaochenyang20]

@finger92 after rerun it, we can merge. No need to do anything on your side.

[minleminzui]

#5054 (comment)

[zhaochenyang20]

@finger92 COuld you please rebase with main, I don't have the right to write. So it fails due to lack of #5503

[finger92]

@finger92 COuld you please rebase with main, I don't have the right to write. So it fails due to lack of #5503

Thanks for replying, I have updated it

[zhaochenyang20]

@finger92 cool, wait for us to run the CI. No need to do anything!

[finger92]

@zhaochenyang20 I saw the CI has been passed, do I need to update branch again?

[minleminzui]

@zhaochenyang20 I saw the CI has been passed, do I need to update branch again?

No need. Just wait for it to be merged

[YCG09]

@finger92 I encountered a problem when using --tool-call-parser deepseekv3 to enable tool calling: it conflicts with Multi-token Prediction (MTP). I followed the documentation from https://docs.sglang.ai/references/deepseek.html regarding MTP usage for DeepSeek models, with a launch command like:

python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3-0324 \
  --speculative-algorithm EAGLE \
  --speculative-draft-model-path lmsys/DeepSeek-V3-0324-NextN \
  --speculative-num-steps 1 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 2 \
  --trust-remote-code \
  --tp 8 

When MTP is enabled, the tool-call-parser fails to return valid tool call results. Disabling MTP allows tool calling to work correctly.
Do you have any idea why enabling MTP conflicts with tool calling in DeepSeek? Is there any way to make them work together?

[minleminzui]

@finger92 I encountered a problem when using --tool-call-parser deepseekv3 to enable tool calling: it conflicts with Multi-token Prediction (MTP). I followed the documentation from https://docs.sglang.ai/references/deepseek.html regarding MTP usage for DeepSeek models, with a launch command like:

python3 -m sglang.launch_server
--model-path deepseek-ai/DeepSeek-V3-0324
--speculative-algorithm EAGLE
--speculative-draft-model-path lmsys/DeepSeek-V3-0324-NextN
--speculative-num-steps 1
--speculative-eagle-topk 1
--speculative-num-draft-tokens 2
--trust-remote-code
--tp 8
When MTP is enabled, the tool-call-parser fails to return valid tool call results. Disabling MTP allows tool calling to work correctly. Do you have any idea why enabling MTP conflicts with tool calling in DeepSeek? Is there any way to make them work together?

please raise an issue， thanks. Our current tool calls are all based on structral tags, which leads to conflicts with speculative decoding

[stevewein]

After deploying deepseekv3-0324 using sglang 0.4.5-post3-cu124, when testing the function call, it was found that if the message contains system information, the function behaves abnormally.

for example

---

curl "http://XXX/v1/chat/completions"
-H "Content-Type: application/json"
-d '{
"messages": [
{
"role": "system",
"content": "Use the tools to answer the questions."
},
{
"role": "user",
"content": "What is the weather in Tokyo"
}
],
"temperature": 0.3,
"max_tokens": 1000,
"model": "mssgpt_v3",
"tool_choice": "auto",
"tools": [
{
"type": "function",
"function": {
"name": "add",
"description": "Add two numbers",
"parameters": {
"properties": {
"a": {
"title": "A",
"type": "integer"
},
"b": {
"title": "B",
"type": "integer"
}
},
"required": [
"a",
"b"
],
"title": "addArguments",
"type": "object"
}
}
},
{
"type": "function",
"function": {
"name": "get_secret_word",
"description": "",
"parameters": {
"properties": {},
"title": "get_secret_wordArguments",
"type": "object"
}
}
},
{
"type": "function",
"function": {
"name": "get_current_weather",
"description": "",
"parameters": {
"properties": {
"city": {
"title": "City",
"type": "string"
}
},
"required": [
"city"
],
"title": "get_current_weatherArguments",
"type": "object"
}
}
}
]
}
'
the function call is wrong and "content" is not null, "tool_calls" is null

{
"id": "1d2854e227fe437191ee78b16e1bcfd8",
"object": "chat.completion",
"created": 1745811416,
"model": "mssgpt-v3",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "I'll add the numbers 7 and 22 for you. One moment.???????????????????????????",
"reasoning_content": null,
"tool_calls": null
},
"logprobs": null,
"finish_reason": "stop",
"matched_stop": 1
}
],
"usage": {
"prompt_tokens": 278,
"total_tokens": 349,
"completion_tokens": 71,
"prompt_tokens_details": null
}
}

---

BUT....

---

curl "http://xxxx/v1/chat/completions"
-H "Content-Type: application/json"
-d '{
"messages": [

    {
        "role": "user",
        "content": "What is the weather in Tokyo"
    }
],
"temperature": 0.3,
"max_tokens": 1000,
"model": "mssgpt_v3",
"tool_choice": "auto",
"tools": [
    {
        "type": "function",
        "function": {
            "name": "add",
            "description": "Add two numbers",
            "parameters": {
                "properties": {
                    "a": {
                        "title": "A",
                        "type": "integer"
                    },
                    "b": {
                        "title": "B",
                        "type": "integer"
                    }
                },
                "required": [
                    "a",
                    "b"
                ],
                "title": "addArguments",
                "type": "object"
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "get_secret_word",
            "description": "",
            "parameters": {
                "properties": {},
                "title": "get_secret_wordArguments",
                "type": "object"
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "",
            "parameters": {
                "properties": {
                    "city": {
                        "title": "City",
                        "type": "string"
                    }
                },
                "required": [
                    "city"
                ],
                "title": "get_current_weatherArguments",
                "type": "object"
            }
        }
    }
]

}
'
function call correct tool_calls reuslt successs

{
"id": "f387b0fe9e5a40e8b40a585d69dcfeb1",
"object": "chat.completion",
"created": 1745811225,
"model": "mssgpt_v3",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": null,
"reasoning_content": null,
"tool_calls": [
{
"id": "2",
"type": "function",
"function": {
"name": "get_current_weather",
"arguments": "{"city": "Tokyo"}"
}
}
]
},
"logprobs": null,
"finish_reason": "tool_calls",
"matched_stop": null
}
],
"usage": {
"prompt_tokens": 261,
"total_tokens": 282,
"completion_tokens": 21,
"prompt_tokens_details": null
}
}

[stevewein]

@finger92 After deploying deepseekv3-0324 using sglang 0.4.5-post3-cu124,
1、when testing the function call, it was found that if the message contains system information, the function behaves abnormally.
2、when testing the function call with message of role "tool",
the sglang replay with " Internal Server Error "

[stevewein]

@finger92 please check issue #5814 #5815

[lambert0312]

1、when testing the function call, it was found that if the message contains system information, the function behaves abnormally.

I think it's because of the following code:

                if (
                    tools
                    and tokenizer_manager.server_args.tool_call_parser == "deepseekv3"
                ):
                    # add function call prompt to deepseekv3
                    openai_compatible_messages.append(
                        {
                            "role": "system",
                            "content": """You are a helpful Assistant.
                    ## Tools
                    ### Function
                    You have the following functions available:
                    """
                            + "".join(
                                [
                                    f"""
                        - `{tool['name']}`:
                        ```json
                        {json.dumps(tool)}
                        ```
                        """
                                    for tool in tools
                                ]
                            ),
                        }
                    )

If the user requests to add his own system prompt words, it will conflict with this!

[ForcewithMe66]

Hi @stevewein , I also met the problems

when testing the function call with message of role "tool",the sglang replay with " Internal Server Error "

I can fix this problem by setting --disable-cuda-graph as suggested in the comment sgl code.

However, the TPS drops from 50 to 10 on 1 H20(141G) nodes. @minleminzui .

[finger92]

Hi @stevewein , I also met the problems

when testing the function call with message of role "tool",the sglang replay with " Internal Server Error "

I can fix this problem by setting --disable-cuda-graph as suggested in the comment sgl code.

However, the TPS drops from 50 to 10 on 1 H20(141G) nodes. @minleminzui .

I trid with the latest sglang code also on 1 H20 node, and the tps is about 50t/s, my command is

python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-V3-0324 --tp 8 --trust-remote-code --mem-fraction-static 0.9 --tool-call-parser deepseekv3

[CatherineSue]

why do we not add a chat template for this? this kind of prompt engineering is more suitable to add into a chat template.

[lambert0312]

why do we not add a chat template for this? this kind of prompt engineering is more suitable to add into a chat template.

Agree to update a new built-in template

[lambert0312]

why do we not add a chat template for this? this kind of prompt engineering is more suitable to add into a chat template.

Agree to update a new built-in template

[finger92]

Got it, we can create a new PR by using custom chat template