Introduce moe_dense_tp_size to fix dense layer errors in DeepSeek V3 + 4x8xH100

[fzyzcjy]

This code depends on #5190 and #5205 and please subtract diff from there

Speed tests

4x8xH100 cannot run without this PR, thus 2x8xH100 is tested.

Before PR:

python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3-0324 --trust-remote-code --tp 16 --dp 16 --enable-dp-attention --enable-deepep-moe --disable-cuda-graph --enable-flashmla --disable-radix-cache --decode-log-interval 1 --host 0.0.0.0 --port 33221 --nnodes 2 --dist-init-addr 10.10.38.10:23456 --chunked-prefill-size 16384 --node-rank 0

Run bench_serving 5 times and get Total token throughput (tok/s) as:

• 15715
• 19388
• 19873
• 20031
• 19922

After PR:

python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3-0324 --trust-remote-code --tp 16 --dp 16 --enable-dp-attention --enable-deepep-moe --disable-cuda-graph --enable-flashmla --disable-radix-cache --decode-log-interval 1 --host 0.0.0.0 --port 33445 --nnodes 2 --dist-init-addr 10.10.38.10:34567 --chunked-prefill-size 16384 --node-rank 0 --moe-dense-tp-size 1

Results

• 7900
• 19884
• 20083
• 20318
• 17505

GSM8K: 93.8

benchmark and test commands

python3 -m sglang.bench_serving --backend sglang --port 32123 --dataset-name random --num-prompt 1024 --random-input 1000 --random-output 1 --random-range-ratio 1 --max-concurrency 1024 --host 10.10.38.10 --port 33221

(cd /host_home/primary_synced/sglang && python3 benchmark/gsm8k/bench_sglang.py --port 20000 --parallel 1400 --num-questions 1400)

UPDATE: more gsm8k tests

# baseline
python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3-0324 --trust-remote-code --tp 16 --dp 16 --enable-dp-attention --enable-deepep-moe --deepep-mode normal --disable-cuda-graph --enable-flashmla --disable-radix-cache --decode-log-interval 1 --host 0.0.0.0 --port 20000 --nnodes 2 --dist-init-addr 10.10.38.10:23456 --chunked-prefill-size 16384 --node-rank 0

# PR
python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3-0324 --trust-remote-code --tp 16 --dp 16 --enable-dp-attention --enable-deepep-moe --deepep-mode normal --disable-cuda-graph --enable-flashmla --disable-radix-cache --decode-log-interval 1 --host 0.0.0.0 --port 20000 --nnodes 2 --dist-init-addr 10.10.38.10:23456 --chunked-prefill-size 16384 --moe-dense-tp-size 1 --node-rank 0

# test
(cd /host_home/primary_synced/sglang && python3 benchmark/gsm8k/bench_sglang.py --port 20000 --parallel 1400 --num-questions 1400)

# baseline outputs
Accuracy: 0.950
Accuracy: 0.942
Accuracy: 0.938
Accuracy: 0.932
Accuracy: 0.936
Accuracy: 0.942
Accuracy: 0.935
Accuracy: 0.945
Accuracy: 0.942
Accuracy: 0.938

# PR outputs
Accuracy: 0.933
Accuracy: 0.943
Accuracy: 0.935
Accuracy: 0.939
Accuracy: 0.940
Accuracy: 0.943
Accuracy: 0.933
Accuracy: 0.941
Accuracy: 0.938
Accuracy: 0.932

baseline: mean=0.940, std=0.005
pr: mean=0.938, std=0.004

Motivation

Original error:

  File "/host_home/primary_synced/sglang/python/sglang/srt/managers/scheduler.py", line 1993, in run_scheduler_process
    scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, dp_rank)
  File "/host_home/primary_synced/sglang/python/sglang/srt/managers/scheduler.py", line 249, in __init__
    self.tp_worker = TpWorkerClass(
  File "/host_home/primary_synced/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 63, in __init__
    self.worker = TpModelWorker(server_args, gpu_id, tp_rank, dp_rank, nccl_port)
  File "/host_home/primary_synced/sglang/python/sglang/srt/managers/tp_worker.py", line 74, in __init__
    self.model_runner = ModelRunner(
  File "/host_home/primary_synced/sglang/python/sglang/srt/model_executor/model_runner.py", line 169, in __init__
    self.initialize(min_per_gpu_memory)
  File "/host_home/primary_synced/sglang/python/sglang/srt/model_executor/model_runner.py", line 179, in initialize
    self.load_model()
  File "/host_home/primary_synced/sglang/python/sglang/srt/model_executor/model_runner.py", line 388, in load_model
    self.model = get_model(
  File "/host_home/primary_synced/sglang/python/sglang/srt/model_loader/__init__.py", line 22, in get_model
    return loader.load_model(
  File "/host_home/primary_synced/sglang/python/sglang/srt/model_loader/loader.py", line 365, in load_model
    model = _initialize_model(
  File "/host_home/primary_synced/sglang/python/sglang/srt/model_loader/loader.py", line 146, in _initialize_model
    return model_class(
  File "/host_home/primary_synced/sglang/python/sglang/srt/models/deepseek_v2.py", line 1308, in __init__
    self.model = DeepseekV2Model(
  File "/host_home/primary_synced/sglang/python/sglang/srt/models/deepseek_v2.py", line 1258, in __init__
    [
  File "/host_home/primary_synced/sglang/python/sglang/srt/models/deepseek_v2.py", line 1259, in <listcomp>
    DeepseekV2DecoderLayer(
  File "/host_home/primary_synced/sglang/python/sglang/srt/models/deepseek_v2.py", line 1046, in __init__
    self.mlp = DeepseekV2MLP(
  File "/host_home/primary_synced/sglang/python/sglang/srt/models/deepseek_v2.py", line 102, in __init__
    self.gate_up_proj = MergedColumnParallelLinear(
  File "/host_home/primary_synced/sglang/python/sglang/srt/layers/linear.py", line 506, in __init__
    super().__init__(
  File "/host_home/primary_synced/sglang/python/sglang/srt/layers/linear.py", line 361, in __init__
    self.quant_method.create_weights(
  File "/host_home/primary_synced/sglang/python/sglang/srt/layers/quantization/fp8.py", line 238, in create_weights
    raise ValueError(
ValueError: Weight output_partition_size = 576 is not divisible by weight quantization block_n = 128.

Modifications

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[zhangml]

@fzyzcjy In the 4x8xH800 environment, when dp=8 and tp=32, the following error occurs:

File "/sgl-workspace/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 267, in init
self.capture()
File "/sgl-workspace/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 351, in capture
) = self.capture_one_batch_size(bs, forward)
File "/sgl-workspace/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 443, in capture_one_batch_size
run_once()
File "/sgl-workspace/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 436, in run_once
logits_output = forward(input_ids, forward_batch.positions, forward_batch)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 1426, in forward
hidden_states = self.model(input_ids, positions, forward_batch, input_embeds)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 1359, in forward
hidden_states, residual = layer(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 1154, in forward
return self.forward_mode_mlp_input_one(
File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 1270, in forward_mode_mlp_input_one
hidden_states, residual = self.post_attention_layernorm(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/sgl-workspace/sglang/python/sglang/srt/custom_op.py", line 18, in forward
return self._forward_method(*args, **kwargs)
File "/sgl-workspace/sglang/python/sglang/srt/layers/layernorm.py", line 56, in forward_cuda
fused_add_rmsnorm(x, residual, self.weight.data, self.variance_epsilon)
File "/usr/local/lib/python3.10/dist-packages/sgl_kernel/elementwise.py", line 24, in fused_add_rmsnorm
torch.ops.sgl_kernel.fused_add_rmsnorm.default(input, residual, weight, eps)
File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 716, in call
return self._op(*args, **kwargs)
RuntimeError: CHECK_EQ(input.size(0), residual.size(0)) failed. 32 vs 128

[fzyzcjy]

@zhangml hi, could you please show full command? this PR solves specific problem that MLP layer gets too fragmented for some scenarios, but there may be other errors if you use other commands.

[zhangml]

@zhangml hi, could you please show full command? this PR solves specific problem that MLP layer gets too fragmented for some scenarios, but there may be other errors if you use other commands.

I use the following command in the 4x8xH800 environment.

python3 -m sglang.launch_server --model-path /code/llm-benchmark-script/data/raw/DeepSeek-V3 --host 0.0.0.0 --port 6178 --tp 32 --dp 8 --enable-dp-attention --disable-radix-cache --trust-remote-code --chunked-prefill-size 4096 --enable-deepep-moe --max-running-requests 128 --disable-radix-cache --mem-fraction-static0.8 --stream-output --deepep-mode low_latency --cuda-graph-max-bs 128 --dist-init-addr xxxxx:20000 --nnodes 4 --node-rank 1 --moe-dense-tp-size 1

[fzyzcjy]

--tp 32 --dp 8 hmm not sure whether this part works or not, I only tested --tp 32 --dp 32 before and was most interested in it because it mimics deepseek's blog post.

[zhangml]

--tp 32 --dp 8 hmm not sure whether this part works or not, I only tested --tp 32 --dp 32 before and was most interested in it because it mimics deepseek's blog post.

Thank you for your answer. The configuration used by the 4 nodes in the DeepSeek paper is dp=8 tp=4, which is equivalent to sglang's dp=8 tp=32. I measured that the single-step delay of dp32 is too large.

[fzyzcjy]

Ah I see, I was talking about deepseek open source week's blogpost, which seems to be DP32. Anyway I may not have time now for that case, but feel free to PR

[CSEEduanyu]

Great job, it looks like this could solve the problem I'm facing.(#5405) Have you tested it on the EP64 model?

[CSEEduanyu]

Currently, setting tp=1 will redundantly copy the weights of the first three layers of the dense mlp 32 times. Is my understanding correct?

[fzyzcjy]

will redundantly copy the weights of the first three layers of the dense mlp 32 times. Is my understanding correct?

Yes, I remembered (very blurly) that deepseek also does this

[CSEEduanyu]

will redundantly copy the weights of the first three layers of the dense mlp 32 times. Is my understanding correct?

Yes, I remembered (very blurly) that deepseek also does this

will redundantly copy the weights of the first three layers of the dense mlp 32 times. Is my understanding correct?

Yes, I remembered (very blurly) that deepseek also does this

Hello, may I ask when this PR will be merged into the master branch?

[fzyzcjy]

GSM8k is normal (roughly 93 percent) on the last merge commit

[CSEEduanyu]

python3 -m sglang.launch_server --model-path DeepSeek-V3-0324/ --tp-size 32 --nccl-init-addr 10.105.216.78:20000 --nnodes 4 --node-rank $RANK --trust-remote-code --host 0.0.0.0 - -schedule-policy fcfs --chunked-prefill-size 65536 --decode-log-interval 1 --disable-cuda-graph --enable-dp-attention --dp 32 --mem-fraction-static 0.76 --disable-radix-cache --moe-dense-tp-size 1

[2025-04-27 08:21:28 DP8 TP8] Scheduler hit an exception: Traceback (most recent call last):
File "/python/sglang/srt/managers/scheduler.py", line 2057, in run_scheduler_process
scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, dp_rank)
File "/python/sglang/srt/managers/scheduler.py", line 260, in init
self.tp_worker = TpWorkerClass(
File "/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 63, in init
self.worker = TpModelWorker(server_args, gpu_id, tp_rank, dp_rank, nccl_port)
File "/python/sglang/srt/managers/tp_worker.py", line 75, in init
self.model_runner = ModelRunner(
File "/python/sglang/srt/model_executor/model_runner.py", line 185, in init
self.initialize(min_per_gpu_memory)
File "/python/sglang/srt/model_executor/model_runner.py", line 195, in initialize
self.load_model()
File "/python/sglang/srt/model_executor/model_runner.py", line 425, in load_model
self.model = get_model(
File "/python/sglang/srt/model_loader/init.py", line 22, in get_model
return loader.load_model(
File "/python/sglang/srt/model_loader/loader.py", line 372, in load_model
model = _initialize_model(
File "/python/sglang/srt/model_loader/loader.py", line 153, in _initialize_model
return model_class(
File "/python/sglang/srt/models/deepseek_v2.py", line 1402, in init
self.model = DeepseekV2Model(
File "/python/sglang/srt/models/deepseek_v2.py", line 1328, in init
[
File "/python/sglang/srt/models/deepseek_v2.py", line 1329, in
DeepseekV2DecoderLayer(
File "/python/sglang/srt/models/deepseek_v2.py", line 1096, in init
self.mlp = DeepseekV2MoE(
File "/python/sglang/srt/models/deepseek_v2.py", line 207, in init
self.experts = MoEImpl(
File "/python/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 324, in init
self.quant_method.create_weights(
File "/python/sglang/srt/layers/quantization/fp8.py", line 504, in create_weights
raise ValueError(
ValueError: The output_size of gate's and up's weight = 64 is not divisible by weight quantization block_n = 128.
@fzyzcjy Just enabling DP and TP doesn't seem to be enough to start it?

[fzyzcjy]

Looks like you are using FusedMoE across 4 nodes and that has error. My fix is to let dense layer support it which is an orthogonal issue.

[GGKOP]

python3 -m sglang.launch_server --model-path DeepSeek-V3-0324/ --tp-size 32 --nccl-init-addr 10.105.216.78:20000 --nnodes 4 --node-rank $RANK --trust-remote-code --host 0.0.0.0 - -schedule-policy fcfs --chunked-prefill-size 65536 --decode-log-interval 1 --disable-cuda-graph --enable-dp-attention --dp 32 --mem-fraction-static 0.76 --disable-radix-cache --moe-dense-tp-size 1

[2025-04-27 08:21:28 DP8 TP8] Scheduler hit an exception: Traceback (most recent call last): File "/python/sglang/srt/managers/scheduler.py", line 2057, in run_scheduler_process scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, dp_rank) File "/python/sglang/srt/managers/scheduler.py", line 260, in init self.tp_worker = TpWorkerClass( File "/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 63, in init self.worker = TpModelWorker(server_args, gpu_id, tp_rank, dp_rank, nccl_port) File "/python/sglang/srt/managers/tp_worker.py", line 75, in init self.model_runner = ModelRunner( File "/python/sglang/srt/model_executor/model_runner.py", line 185, in init self.initialize(min_per_gpu_memory) File "/python/sglang/srt/model_executor/model_runner.py", line 195, in initialize self.load_model() File "/python/sglang/srt/model_executor/model_runner.py", line 425, in load_model self.model = get_model( File "/python/sglang/srt/model_loader/init.py", line 22, in get_model return loader.load_model( File "/python/sglang/srt/model_loader/loader.py", line 372, in load_model model = _initialize_model( File "/python/sglang/srt/model_loader/loader.py", line 153, in _initialize_model return model_class( File "/python/sglang/srt/models/deepseek_v2.py", line 1402, in init self.model = DeepseekV2Model( File "/python/sglang/srt/models/deepseek_v2.py", line 1328, in init [ File "/python/sglang/srt/models/deepseek_v2.py", line 1329, in DeepseekV2DecoderLayer( File "/python/sglang/srt/models/deepseek_v2.py", line 1096, in init self.mlp = DeepseekV2MoE( File "/python/sglang/srt/models/deepseek_v2.py", line 207, in init self.experts = MoEImpl( File "/python/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 324, in init self.quant_method.create_weights( File "/python/sglang/srt/layers/quantization/fp8.py", line 504, in create_weights raise ValueError( ValueError: The output_size of gate's and up's weight = 64 is not divisible by weight quantization block_n = 128. @fzyzcjy Just enabling DP and TP doesn't seem to be enough to start it?

I encountered the same problem. How did you solve it?

[junliu-mde]

python3 -m sglang.launch_server --model-path DeepSeek-V3-0324/ --tp-size 32 --nccl-init-addr 10.105.216.78:20000 --nnodes 4 --node-rank $RANK --trust-remote-code --host 0.0.0.0 - -schedule-policy fcfs --chunked-prefill-size 65536 --decode-log-interval 1 --disable-cuda-graph --enable-dp-attention --dp 32 --mem-fraction-static 0.76 --disable-radix-cache --moe-dense-tp-size 1
[2025-04-27 08:21:28 DP8 TP8] Scheduler hit an exception: Traceback (most recent call last): File "/python/sglang/srt/managers/scheduler.py", line 2057, in run_scheduler_process scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, dp_rank) File "/python/sglang/srt/managers/scheduler.py", line 260, in init self.tp_worker = TpWorkerClass( File "/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 63, in init self.worker = TpModelWorker(server_args, gpu_id, tp_rank, dp_rank, nccl_port) File "/python/sglang/srt/managers/tp_worker.py", line 75, in init self.model_runner = ModelRunner( File "/python/sglang/srt/model_executor/model_runner.py", line 185, in init self.initialize(min_per_gpu_memory) File "/python/sglang/srt/model_executor/model_runner.py", line 195, in initialize self.load_model() File "/python/sglang/srt/model_executor/model_runner.py", line 425, in load_model self.model = get_model( File "/python/sglang/srt/model_loader/init.py", line 22, in get_model return loader.load_model( File "/python/sglang/srt/model_loader/loader.py", line 372, in load_model model = _initialize_model( File "/python/sglang/srt/model_loader/loader.py", line 153, in _initialize_model return model_class( File "/python/sglang/srt/models/deepseek_v2.py", line 1402, in init self.model = DeepseekV2Model( File "/python/sglang/srt/models/deepseek_v2.py", line 1328, in init [ File "/python/sglang/srt/models/deepseek_v2.py", line 1329, in DeepseekV2DecoderLayer( File "/python/sglang/srt/models/deepseek_v2.py", line 1096, in init self.mlp = DeepseekV2MoE( File "/python/sglang/srt/models/deepseek_v2.py", line 207, in init self.experts = MoEImpl( File "/python/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 324, in init self.quant_method.create_weights( File "/python/sglang/srt/layers/quantization/fp8.py", line 504, in create_weights raise ValueError( ValueError: The output_size of gate's and up's weight = 64 is not divisible by weight quantization block_n = 128. @fzyzcjy Just enabling DP and TP doesn't seem to be enough to start it?

I encountered the same problem. How did you solve it?

I believe current implementation still not support FusedMoE or CUDA graph with TP 32.
--tp 32 --enable-ep-moe --disable-cuda-graph should work, although I think use case of this setting is limited.