[Feature] Support DeepSeek VL 2

[zhyncs]

Motivation

deepseek-vl2 is one of the best vision language models. We would like to support it.

https://huggingface.co/deepseek-ai/deepseek-vl2
https://github.com/deepseek-ai/DeepSeek-VL2

Related resources

You can learn from the existing implementations and usage examples of other vision language models.
https://sgl-project.github.io/references/supported_models.html#how-to-support-a-new-model
https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/llava.py
https://github.com/sgl-project/sglang/blob/main/test/srt/test_vision_openai_server.py
https://sgl-project.github.io/references/sampling_params.html#multi-modal

[Edenzzzz]

It's mainly about replacing linear and attn with sgl modules?

[bao231]

什么时候能支持DeepSeek VL 2 ？

[Edenzzzz]

@zhyncs Would you recommend keeping the prefill logic?
https://github.com/deepseek-ai/DeepSeek-VL2/blob/66ec91081c0eaa7db4212ba34358fa3b1d159f4f/deepseek_vl2/models/modeling_deepseek_vl_v2.py#L480

[ccw1996]

hi, I am interested in contributing to support deepseek vl 2. Can I do for it?

[zhyncs]

@Edenzzzz @ccw1996 Hi all feel free to collaborate with each other. Thanks.

[Edenzzzz]

Sounds good. I plan to look into optimizing cache offload more and then this. @ccw1996 We can perhaps work on a branch together