compressed_tensors: port w8a16 fp8 from vllm

[vhain]

Motivation

Fixes #4851

FP8 W8A8 could be the most popular FP8 quant method when using compressed_tensors with llm-compressor, as it's the one llm-compressor's example uses by default.

However FP8 W8A8 kernel is not supported on machines with CUDA compute capability < 89, and the original logic tries to fallback to use FP8W8A16, which is not ported from vLLM yet, raising following exception:

NameError: name 'CompressedTensorsW8A16Fp8' is not defined. Did you mean: 'CompressedTensorsW8A8Fp8'?

Modifications

This PR ports CompressedTensorsW8A16Fp8 from vLLM.

It lazy imports vllm without introducing hard dependency.

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

cc: @BBuf

[BBuf]

Can you add a unit_test in here ?

[vhain]

@BBuf Two things:

1. By looking at the test code you referenced, it seems like that current test is not testing compressed-tensors FP8 W8A8 but AWQ and GPTQ. Am I understanding it correctly?

sglang/test/srt/test_vllm_dependency.py

Line 119 in 6ffb6bd

(parse_models(DEFAULT_MODEL_NAME_FOR_NIGHTLY_EVAL_QUANT_TP1), False, False),

sglang/python/sglang/test/test_utils.py

Line 62 in 6ffb6bd

DEFAULT_MODEL_NAME_FOR_NIGHTLY_EVAL_QUANT_TP1 = "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4,hugging-quants/Meta-Llama-3.1-8B-Instruct-GPTQ-INT4,hugging-quants/Mixtral-8x7B-Instruct-v0.1-AWQ-INT4"

2. By looking at current if clauses that selects which compressed-tensors scheme to use, running it on machines with CUDA compute capability < 89 is the only way to trigger initiation of CompressedTensorsW8A16Fp8. It seems like vllm dependency test currently only runs on runner with H100s. Are there any runner that we can setup with GPUs with lower compute capacity?

sglang/.github/workflows/vllm-dependency-test.yml

Line 29 in 6ffb6bd

runs-on: 1-gpu-runner

https://github.com/sgl-project/sglang/actions/runs/14125041059/job/39572201534#step:3:22

[BBuf]

@BBuf Two things:

1. By looking at the test code you referenced, it seems like that current test is not testing compressed-tensors FP8 W8A8 but AWQ and GPTQ. Am I understanding it correctly?

sglang/test/srt/test_vllm_dependency.py

Line 119 in 6ffb6bd

(parse_models(DEFAULT_MODEL_NAME_FOR_NIGHTLY_EVAL_QUANT_TP1), False, False),

sglang/python/sglang/test/test_utils.py

Line 62 in 6ffb6bd

DEFAULT_MODEL_NAME_FOR_NIGHTLY_EVAL_QUANT_TP1 = "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4,hugging-quants/Meta-Llama-3.1-8B-Instruct-GPTQ-INT4,hugging-quants/Mixtral-8x7B-Instruct-v0.1-AWQ-INT4"

2. By looking at current if clauses that selects which compressed-tensors scheme to use, running it on machines with CUDA compute capability < 89 is the only way to trigger initiation of CompressedTensorsW8A16Fp8. It seems like vllm dependency test currently only runs on runner with H100s. Are there any runner that we can setup with GPUs with lower compute capacity?

sglang/.github/workflows/vllm-dependency-test.yml

Line 29 in 6ffb6bd

runs-on: 1-gpu-runner

https://github.com/sgl-project/sglang/actions/runs/14125041059/job/39572201534#step:3:22

Ok, make sense. The test file only aims to test quant models which quant method must use vLLM now, compressed-tensors FP8 W8A8 quant format model is supported by SGLang without vllm dependecy now and it's tested in another files.

[BBuf]

LGTM!

[vhain]

@zhyncs Could you PTAL? Compressed tensors FP8 W8A8 in main is currently broken against old CUDA GPUs (compute capability under 89) #4851

[vhain]

@BBuf When can this be merged in? We've been running the forked version with this change on our production and it's been working fine. We'd like to withdraw from forked version asap and deploy the upstream 😃. I've seen 0.4.4.post3 just released but it still have this bug #4851 ..

cc: @zhyncs

[qeternity]

Hi all - would be great to get this merged, quite a few people with local dev setups on older architectures.

[ashwin-js]

Hey can we also add compressed_tensors_w8a8_int8.py ?

[ehartford]

ImportError: vllm is not installed. To use CompressedTensorsW8A16Fp8, please install vllm

that's a bit like linux depending on windows, innit?

[qeternity]

@zhyncs @BBuf hi both - can we please get this merged?

[BBuf]

@zhyncs @BBuf hi both - can we please get this merged?

Yes, we can merge it. cc @zhyncs

[zhyncs]

Please merge the latest main