Description
Checklist
- 1. I have searched related issues but cannot get the expected help.
- 2. The bug has not been fixed in the latest version.
- 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
- 5. Please use English, otherwise it will be closed.
Describe the bug
For Qwen/Qwen2.5-VL-7B-Instruct, after running sglang.bench_serving with specific configurations, TTFT and ITL are not properly reported.
Here is an example report,
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 8
Successful requests: 50
Benchmark duration (s): 91.89
Total input tokens: 1161817
Total generated tokens: 51200
Total generated tokens (retokenized): 0
Request throughput (req/s): 0.54
Input token throughput (tok/s): 12643.08
Output token throughput (tok/s): 557.17
Total token throughput (tok/s): 13200.25
Concurrency: 7.20
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 13238.65
Median E2E Latency (ms): 13254.39
---------------Time to First Token----------------
Mean TTFT (ms): 0.00
Median TTFT (ms): 0.00
P99 TTFT (ms): 0.00
---------------Inter-Token Latency----------------
Mean ITL (ms): 0.00
Median ITL (ms): 0.00
P95 ITL (ms): 0.00
P99 ITL (ms): 0.00
Max ITL (ms): 0.00
==================================================
Reproduction
First, launch SGLang server,
python -m sglang.launch_server --context-length=65536 --trust-remote-code --model-path Qwen/Qwen2.5-VL-7B-Instruct
Then, start the benchmark,
python -m sglang.bench_serving --backend sglang --max-concurrency 8 --dataset-name mmmu --num-prompts 50Environment
Python: 3.12.9 | packaged by Anaconda, Inc. | (main, Feb 6 2025, 18:56:27) [GCC 11.2.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA A100-SXM4-80GB
GPU 0,1,2,3,4,5,6,7 Compute Capability: 8.0
CUDA_HOME: /home/zijzhang/miniconda3/envs/sgl
NVCC: Cuda compilation tools, release 12.8, V12.8.93
CUDA Driver Version: 535.129.03
PyTorch: 2.6.0+cu124
sglang: 0.4.6.post4
sgl_kernel: 0.1.3
flashinfer_python: 0.2.5
triton: 3.2.0
transformers: 4.51.1
torchao: 0.9.0
numpy: 2.2.4
aiohttp: 3.11.14
fastapi: 0.115.12
hf_transfer: 0.1.9
huggingface_hub: 0.30.0
interegular: 0.3.3
modelscope: 1.24.1
orjson: 3.10.16
outlines: 0.1.11
packaging: 24.2
psutil: 7.0.0
pydantic: 2.11.1
python-multipart: 0.0.20
pyzmq: 26.3.0
uvicorn: 0.34.0
uvloop: 0.21.0
vllm: Module Not Found
xgrammar: 0.1.19
openai: 1.69.0
tiktoken: 0.9.0
anthropic: 0.49.0
litellm: 1.65.0
decord: 0.6.0
NVIDIA Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 NIC10 NIC11 NIC12 NIC13 NIC14 NIC15 NIC16NIC17 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV12 NV12 NV12 NV12 NV12 NV12 NV12 SYS SYS SYS SYS SYS PXB PXB PXB PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS 48-63,176-191 3 N/A
GPU1 NV12 X NV12 NV12 NV12 NV12 NV12 NV12 SYS SYS SYS SYS SYS PXB PXB PXB PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS 48-63,176-191 3 N/A
GPU2 NV12 NV12 X NV12 NV12 NV12 NV12 NV12 SYS PXB PXB PXB PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS 16-31,144-159 1 N/A
GPU3 NV12 NV12 NV12 X NV12 NV12 NV12 NV12 SYS PXB PXB PXB PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS 16-31,144-159 1 N/A
GPU4 NV12 NV12 NV12 NV12 X NV12 NV12 NV12 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PXB PXB PXB PXB 112-127,240-254 7 N/A
GPU5 NV12 NV12 NV12 NV12 NV12 X NV12 NV12 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PXB PXB PXB PXB 112-127,240-254 7 N/A
GPU6 NV12 NV12 NV12 NV12 NV12 NV12 X NV12 SYS SYS SYS SYS SYS SYS SYS SYS SYS PXB PXB PXB PXB SYS SYS SYS SYS SYS 80-95,208-223 5 N/A
GPU7 NV12 NV12 NV12 NV12 NV12 NV12 NV12 X SYS SYS SYS SYS SYS SYS SYS SYS SYS PXB PXB PXB PXB SYS SYS SYS SYS SYS 80-95,208-223 5 N/A
NIC0 SYS SYS SYS SYS SYS SYS SYS SYS X SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS
NIC1 SYS SYS PXB PXB SYS SYS SYS SYS SYS X PIX PXB PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS
NIC2 SYS SYS PXB PXB SYS SYS SYS SYS SYS PIX X PXB PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS
NIC3 SYS SYS PXB PXB SYS SYS SYS SYS SYS PXB PXB X PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS
NIC4 SYS SYS PXB PXB SYS SYS SYS SYS SYS PXB PXB PIX X SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS
NIC5 PXB PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS X PIX PXB PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS
NIC6 PXB PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX X PXB PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS
NIC7 PXB PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PXB PXB X PIX SYS SYS SYS SYS SYS SYS SYS SYS SYS
NIC8 PXB PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PXB PXB PIX X SYS SYS SYS SYS SYS SYS SYS SYS SYS
NIC9 SYS SYS SYS SYS SYS SYS PXB PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS X PIX PXB PXB SYS SYS SYS SYS SYS
NIC10 SYS SYS SYS SYS SYS SYS PXB PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX X PXB PXB SYS SYS SYS SYS SYS
NIC11 SYS SYS SYS SYS SYS SYS PXB PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS PXB PXB X PIX SYS SYS SYS SYS SYS
NIC12 SYS SYS SYS SYS SYS SYS PXB PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS PXB PXB PIX X SYS SYS SYS SYS SYS
NIC13 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS X SYS SYS SYS SYS
NIC14 SYS SYS SYS SYS PXB PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS X PIX PXB PXB
NIC15 SYS SYS SYS SYS PXB PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX X PXB PXB
NIC16 SYS SYS SYS SYS PXB PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PXB PXB X PIX
NIC17 SYS SYS SYS SYS PXB PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PXB PXB PIX X
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
NIC Legend:
NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2
NIC3: mlx5_3
NIC4: mlx5_4
NIC5: mlx5_5
NIC6: mlx5_6
NIC7: mlx5_7
NIC8: mlx5_8
NIC9: mlx5_9
NIC10: mlx5_10
NIC11: mlx5_11
NIC12: mlx5_12
NIC13: mlx5_13
NIC14: mlx5_14
NIC15: mlx5_15
NIC16: mlx5_16
NIC17: mlx5_17
ulimit soft: 1310720