[Bug] Llama4 Scout will hit OOM during launch_server on 8 H200

[hebiao064]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 80.04 GiB. GPU 3 has a total capacity of 79.10 GiB of which 20.08 GiB is free. Process 806478 has 59.01 GiB memory in use. Of the allocated memory 56.90 GiB is allocated by PyTorch, and 125.19 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

Reproduction

python3 -m sglang.launch_server --model meta-llama/Llama-4-Scout-17B-16E-Instruct --trust-remote-code --tp 8 --disable-cuda-graph

Environment

latest main of sglang

[hebiao064]

https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct/blob/main/config.json#L28

Scout Model has 10M Context by default, we have to add --context-length to fix it, for example

 python3 -m sglang.launch_server --model meta-llama/Llama-4-Scout-17B-16E-Instruct --trust-remote-code --tp 8 --disable-cuda-graph --context-length 1000000 

[hebiao064]

Change this from max to min could also fix this issue

sglang/python/sglang/srt/model_executor/model_runner.py

Line 736 in 21514ff

max(

[hebiao064]

The calculation of KV Cache's max_num_reqs looks wrong to me:

sglang/python/sglang/srt/model_executor/model_runner.py

Lines 736 to 742 in 21514ff

	max(
	int(
	self.max_total_num_tokens / self.model_config.context_len * 512
	),
	2048,
	),
	4096,

If self.max_total_num_tokens = 4M and self.model_config.context_len = 10M, max_num_reqs will be set as 2048.

It will fail due to OOM.
@merrymercy @zhyncs @CatherineSue FYI

[hebiao064]

Closing this issue now since it was already captured in here: https://docs.sglang.ai/references/llama4.html#configuration-tips

[fxmarty-amd]

Why isn't max_num_reqs just server_args.max_running_requests?