model: support gemma-3-it

[mickqian]

Motivation

Support gemma3-it.

FYI, gemma3-1b-it is an llm, gemma3-pt series are not chat models.

Modifications

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[yizhang2077]

I think base image prcessor don't need change?

[zhaochenyang20]

@mickqian @yizhang2077 is this ready to merge? approved?

[yizhang2077]

Typically in PRs when adding models, accuracy is reported, for example, on MMLU implemented in the PR compared to the implementation in transformers, and for multimodal models, some multimodal benchmark. Could you advise how you verified the implementation?

@Swipe4057 we have a mmmu benchmark, and for each vlm model, we need verify this benchmark and compare with transformers implementation. Besides, if transformers supports this model, we suggest add an unittest to compare logits with hf.

[yizhang2077]

@mickqian plz paste mmmu benchmark result here.

[yizhang2077]

I add an issue for keeping track of current VLM models performance in mmmu benchmark. We can update benchmark result here #4456 @mickqian @zhaochenyang20

[yizhang2077]

Why we need add regex here?

[yizhang2077]

@zhaochenyang20 this PR can be merged

[mickqian]

@zhaochenyang20 This is ready. Many thanks

[zhaochenyang20]

@mickqian @yizhang2077 thanks. I will tell lianmin!

[zhaochenyang20]

@Ying1123 hey, ying, this can be merged. it's high-prioritized.

[Swipe4057]

@zhaochenyang20 @mickqian Hello, I've been running Gemma3 27b it model in vllm 0.8.0 and sglang installed from source with original weights on an H100 GPU. My results show that for the same long text-only query, the outputs of the models differ significantly. In vllm, generation proceeds normally, but in sglang, during long generation, it starts to degrade into garbage output and continues indefinitely. This phenomenon occurs with long queries and queries containing code. Could someone else test this as well?

Also, a question: is prefix caching supported in sglang for multimodal models, particularly Gemma 3?

[zhaochenyang20]

@zhaochenyang20 @mickqian Hello, I've been running Gemma3 27b it model in vllm 0.8.0 and sglang installed from source with original weights on an H100 GPU. My results show that for the same long text-only query, the outputs of the models differ significantly. In vllm, generation proceeds normally, but in sglang, during long generation, it starts to degrade into garbage output and continues indefinitely. This phenomenon occurs with long queries and queries containing code. Could someone else test this as well?

Also, a question: is prefix caching supported in sglang for multimodal models, particularly Gemma 3?

supported. could you give us your reproducable scripts. We will fix this ASAP>

[Swipe4057]

@zhaochenyang20
Here's my launch command:

Then ask a simple question:
How would you advise fixing high vulnerabilities in a Docker container that are found in the base Debian image?
Temperature 1 and top_p 0.95

You'll see that the generation doesn't stop and something like this will begin:

In the logs, the generation will continue:

[zhaochenyang20]

@zhaochenyang20 Here's my launch command: Then ask a simple question: How would you advise fixing high vulnerabilities in a Docker container that are found in the base Debian image? Temperature 1 and top_p 0.95

You'll see that the generation doesn't stop and something like this will begin:

In the logs, the generation will continue:

cc @mickqian mick

[mickqian]

a fix is on the way

[rangehow]

Gemma3's generation speed is surprisingly slow compared to other 3B/4B models like Qwen2.5-3B. Is the current Gemma3 implementation correct?

[zhaochenyang20]

great!