FP4 weight loading and inference (2/2)

[trevor-m]

Motivation

Requires #3899 for new kernels in sgl-kernel.

This PR is part 2 of 2 to adds support for modelopt FP4 quantized models.
Tested using fp4 quantized Llama 3.1 model.

This work was adapted from the following - thanks @pavanimajety @kaixih @kushanam!
vllm-project/vllm#12784
vllm-project/vllm#13571
vllm-project/vllm#12520

Modifications

Adds modelopt_fp4 quantization method.
Adds ModelOptFp4Config and ModelOptFp4LinearMethod to utilize new fp4 kernels for linear layers

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[zhyncs]

Hi @trevor-m Can you help resolve the conflicts? Thanks. cc @kushanam @elfiegg

[pavanimajety]

@zhyncs The conflicts should automatically resolve once the part 1 of the PR is merged and this PR is rebased on top of that.

[zhyncs]

Hi @trevor-m @kushanam @elfiegg May you help fix this PR's conflicts?

[zhyncs]

@pavanimajety

[trevor-m]

@pavanimajety

Sorry about that, just fixed! @zhyncs

[pavanimajety]

Thanks @trevor-m and @zhyncs! I missed there was a code conflict.

[Edwardf0t1]

Hi @trevor-m @pavanimajety @zhyncs Great work! could we resolve conflicts and get this merged? Thanks.

[trevor-m]

Hi @trevor-m @pavanimajety @zhyncs Great work! could we resolve conflicts and get this merged? Thanks.

Thanks @Edwardf0t1, fixed.

[zhyncs]

remove this

[zhyncs]

May you provide the e2e gsm8k result for nvidia/DeepSeek-R1-FP4

python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1319 --parallel 1319

[Edwardf0t1]

May you provide the e2e gsm8k result for nvidia/DeepSeek-R1-FP4

python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1319 --parallel 1319

@zhyncs I don't think this PR would work for DS R1 FP4 since that requires FP4 MoE kernels.

@trevor-m could you run a quick test and show the outputs for the following:

import sglang as sgl

def main():

    prompts = [
        "Hello, my name is",
        "The president of the United States is",
        "The capital of France is",
        "The future of AI is",
    ]
    sampling_params = {"temperature": 0.8, "top_p": 0.95}
    llm = sgl.Engine(model_path="nvidia/Llama-3.3-70B-Instruct-FP4", quantization="modelopt_fp4")

    outputs = llm.generate(prompts, sampling_params)
    for prompt, output in zip(prompts, outputs):
        print("===============================")
        print(f"Prompt: {prompt}\nGenerated text: {output['text']}")

if __name__ == "__main__":
    main()

[trevor-m]

May you provide the e2e gsm8k result for nvidia/DeepSeek-R1-FP4

python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1319 --parallel 1319

As @Edwardf0t1 mentioned we can't run DS R1 FP4 yet. But I ran your benchmark commands for nvidia/Llama-3.3-70B-Instruct-FP4:

Accuracy: 0.939
Invalid: 0.000
Latency: 65.557 s
Output throughput: 2229.165 token/s

[trevor-m]

@trevor-m could you run a quick test and show the outputs for the following:

import sglang as sgl

def main():

    prompts = [
        "Hello, my name is",
        "The president of the United States is",
        "The capital of France is",
        "The future of AI is",
    ]
    sampling_params = {"temperature": 0.8, "top_p": 0.95}
    llm = sgl.Engine(model_path="nvidia/Llama-3.3-70B-Instruct-FP4", quantization="modelopt_fp4")

    outputs = llm.generate(prompts, sampling_params)
    for prompt, output in zip(prompts, outputs):
        print("===============================")
        print(f"Prompt: {prompt}\nGenerated text: {output['text']}")

if __name__ == "__main__":
    main()

Results

===============================
Prompt: Hello, my name is
Generated text:  Angelique. I am a German Shorthaired Pointer. I am a very happy and energetic dog. I love playing with my toys and going for walks with my human family. I also love swimming and running around in the backyard. My favorite thing to do is play fetch with my favorite ball. I will do just about anything for a belly rub. I am a very loyal and loving companion and I promise to always be by your side. I hope you like my picture and that you will come to visit me soon. I get along great with other dogs and I love people. I am ready to go to my forever home and be
===============================
Prompt: The president of the United States is
Generated text:  just one of many important leaders in the country. The U.S. government has three branches: the executive, legislative, and judicial. The legislative branch is further divided into the Senate and the House of Representatives. The president is the head of the executive branch, but they don't have absolute power. They must work with Congress to pass laws and make decisions for the country.
A. the executive branch of the U.S. government
B. the legislative branch of the U.S. government
C. the judicial branch of the U.S. government
D. the head of the Senate
E. the head of the House of Representatives
===============================
Prompt: The capital of France is
Generated text:  famous for its stunning architecture, art museums, and romantic atmosphere. But there's more to Paris than just the Eiffel Tower and the Louvre. This City Guide will take you on a journey through the city's charming neighborhoods, historic landmarks, and cultural attractions, helping you to discover the authentic Paris that locals love.
We'll start with the basics: when to go, how to get around, and where to stay. Then, we'll dive into the city's must-see attractions, from the iconic Notre-Dame Cathedral to the trendy Marais neighborhood. You'll learn about the city's rich history, from the French Revolution
===============================
Prompt: The future of AI is
Generated text:  not just about the technology, but about the people who will shape it. At Mosaic Smart Data, we believe that the best results come from collaboration between humans and AI systems. Our people are experts in data science, technology, and business, and we work together to deliver cutting-edge AI solutions that drive real value for our clients.
Our team has a passion for innovation and a drive to make a real difference in the financial services industry. We believe in empowering our people to take ownership and to continuously learn and develop their skills. If you are excited about the potential of AI to transform the financial services industry and want to be part of a

[zhyncs]

Great work! @trevor-m @Edwardf0t1

[Graham1025]

@trevor-m could you run a quick test and show the outputs for the following:

import sglang as sgl

def main():

    prompts = [
        "Hello, my name is",
        "The president of the United States is",
        "The capital of France is",
        "The future of AI is",
    ]
    sampling_params = {"temperature": 0.8, "top_p": 0.95}
    llm = sgl.Engine(model_path="nvidia/Llama-3.3-70B-Instruct-FP4", quantization="modelopt_fp4")

    outputs = llm.generate(prompts, sampling_params)
    for prompt, output in zip(prompts, outputs):
        print("===============================")
        print(f"Prompt: {prompt}\nGenerated text: {output['text']}")

if __name__ == "__main__":
    main()

Results

===============================
Prompt: Hello, my name is
Generated text:  Angelique. I am a German Shorthaired Pointer. I am a very happy and energetic dog. I love playing with my toys and going for walks with my human family. I also love swimming and running around in the backyard. My favorite thing to do is play fetch with my favorite ball. I will do just about anything for a belly rub. I am a very loyal and loving companion and I promise to always be by your side. I hope you like my picture and that you will come to visit me soon. I get along great with other dogs and I love people. I am ready to go to my forever home and be
===============================
Prompt: The president of the United States is
Generated text:  just one of many important leaders in the country. The U.S. government has three branches: the executive, legislative, and judicial. The legislative branch is further divided into the Senate and the House of Representatives. The president is the head of the executive branch, but they don't have absolute power. They must work with Congress to pass laws and make decisions for the country.
A. the executive branch of the U.S. government
B. the legislative branch of the U.S. government
C. the judicial branch of the U.S. government
D. the head of the Senate
E. the head of the House of Representatives
===============================
Prompt: The capital of France is
Generated text:  famous for its stunning architecture, art museums, and romantic atmosphere. But there's more to Paris than just the Eiffel Tower and the Louvre. This City Guide will take you on a journey through the city's charming neighborhoods, historic landmarks, and cultural attractions, helping you to discover the authentic Paris that locals love.
We'll start with the basics: when to go, how to get around, and where to stay. Then, we'll dive into the city's must-see attractions, from the iconic Notre-Dame Cathedral to the trendy Marais neighborhood. You'll learn about the city's rich history, from the French Revolution
===============================
Prompt: The future of AI is
Generated text:  not just about the technology, but about the people who will shape it. At Mosaic Smart Data, we believe that the best results come from collaboration between humans and AI systems. Our people are experts in data science, technology, and business, and we work together to deliver cutting-edge AI solutions that drive real value for our clients.
Our team has a passion for innovation and a drive to make a real difference in the financial services industry. We believe in empowering our people to take ownership and to continuously learn and develop their skills. If you are excited about the potential of AI to transform the financial services industry and want to be part of a

@trevor-m Thanks for your great work! But on my 5090 device, I got follow error:
Traceback (most recent call last):
File "/home/lixiang/guihao/lpex-sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 307, in init
self.capture()
File "/home/lixiang/guihao/lpex-sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 386, in capture
) = self.capture_one_batch_size(bs, forward)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lixiang/guihao/lpex-sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 512, in capture_one_batch_size
with torch.cuda.graph(graph, pool=global_graph_memory_pool, stream=stream):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lixiang/lixinlei/envir/yes/envs/guihao-fp4/lib/python3.12/site-packages/torch/cuda/graphs.py", line 186, in exit
self.cuda_graph.capture_end()
File "/home/lixiang/lixinlei/envir/yes/envs/guihao-fp4/lib/python3.12/site-packages/torch/cuda/graphs.py", line 84, in capture_end
super().capture_end()
RuntimeError: CUDA error: operation failed due to a previous error during capture
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/lixiang/guihao/lpex-sglang/python/sglang/srt/managers/scheduler.py", line 2297, in run_scheduler_process
scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, pp_rank, dp_rank)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lixiang/guihao/lpex-sglang/python/sglang/srt/managers/scheduler.py", line 277, in init
self.tp_worker = TpWorkerClass(
^^^^^^^^^^^^^^
File "/home/lixiang/guihao/lpex-sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 64, in init
self.worker = TpModelWorker(
^^^^^^^^^^^^^^
File "/home/lixiang/guihao/lpex-sglang/python/sglang/srt/managers/tp_worker.py", line 78, in init
self.model_runner = ModelRunner(
^^^^^^^^^^^^
File "/home/lixiang/guihao/lpex-sglang/python/sglang/srt/model_executor/model_runner.py", line 231, in init
self.initialize(min_per_gpu_memory)
File "/home/lixiang/guihao/lpex-sglang/python/sglang/srt/model_executor/model_runner.py", line 305, in initialize
self.init_cuda_graphs()
File "/home/lixiang/guihao/lpex-sglang/python/sglang/srt/model_executor/model_runner.py", line 1098, in init_cuda_graphs
self.cuda_graph_runner = CudaGraphRunner(self)
^^^^^^^^^^^^^^^^^^^^^
File "/home/lixiang/guihao/lpex-sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 309, in init
raise Exception(
Exception: Capture CUDA graph failed: CUDA error: operation failed due to a previous error during capture
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Possible solutions:

1. set --mem-fraction-static to a smaller value (e.g., 0.8 or 0.7)
2. set --cuda-graph-max-bs to a smaller value (e.g., 16)
3. disable torch compile by not using --enable-torch-compile
4. disable CUDA graph by --disable-cuda-graph. (Not recommended. Huge performance loss)
   Open an issue on GitHub https://github.com/sgl-project/sglang/issues/new/choose

How to fix this error？ Thanks
Disable cuda graph can fix it, but with low performance