[Model][VLM] Add Qwen2.5-Omni model support (thinker only)

[fyabc]

This PR adding support for Qwen2.5-Omni model (thinker only).

Requirements

This PR requires this corresponding transformers PR.

pip install git+https://github.com/huggingface/transformers@f742a644ca32e65758c3adb36225aef1731bd2a8

Note: You need to install transformers from source from that branch

Example Usage

# Audio + image + video
python examples/offline_inference/qwen2_5_omni/only_thinker.py -q mixed_modalities

# Read vision and audio inputs from a single video file
# NOTE: V1 engine does not support interleaved modalities yet.
VLLM_USE_V1=0 python examples/offline_inference/qwen2_5_omni/only_thinker.py -q use_audio_in_video

# Process audio inputs
python examples/offline_inference/audio_language.py --model-type qwen2_5_omni

# Process image inputs
python examples/offline_inference/vision_language.py --modality image --model-type qwen2_5_omni

# Process video inputs
python examples/offline_inference/vision_language.py --modality video --model-type qwen2_5_omni

Notes

The whole Qwen2.5-Omni model includes three parts:

• thinker: multimodal inputs -> text responses & hidden states
• talker: text responses & hidden states from thinker -> speech codes
• code2wav (streaming codec decoder): codes -> speech

This PR only implements the thinker part now, it accepts multimodal inputs (images / videos / audios), and generate text responses, similar to other common VLMs.
We have also develped an end-to-end implementation (will be released soon), but due to its significant impact on the vLLM framework architecture, we will not create the related pull request for now.

FIX #15563

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[DarkLight1337]

Sorry I don't have time to review in detail tonight, but from a quick glance, can you add this model to the following pages?

• Supported Models page
• tests/models/registry.py (set is_available_online=False to pass CI until the model repo is released on HF)
• tests/models/multimodal/processing/test_common.py
• tests/models/decoder_only/vision_language/test_models.py (optional for now)

[fyabc]

Sorry I don't have time to review in detail tonight, but from a quick glance, can you add this model to the following pages?

• Supported Models page
• tests/models/registry.py (set is_available_online=False to pass CI until the model repo is released on HF)
• tests/models/multimodal/processing/test_common.py
• tests/models/decoder_only/vision_language/test_models.py (optional for now)

OK，I will add them tomorrow.

[yangninghua]

@fyabc Qwen/Qwen2.5-Omni-7B ??

[ywang96]

Sorry for the delay - going to take a look at this PR tonight!

[ywang96]

Thank you for the contribution! I have left some comments!

[fyabc]

Hi @ywang96 @DarkLight1337 , I update some other examples here, please check the code.

[DarkLight1337]

Can you resolve the failures in the basic models test?

[fyabc]

Can you resolve the failures in the basic models test?

Hi @DarkLight1337, I have fixed the test registry, now the api timeout error seems raised outside of thie PR.

[ywang96]

Very sorry for the long delay - let's get this in!