[Bug]: Ernie45-21B and Qwen3-30B have garbled text when start CUDA graph, main branch

[CSWYF3634076]

Your current environment

The output of python collect_env.py

Your output of `python collect_env.py` here

🐛 Describe the bug

# SPDX-License-Identifier: Apache-2.0

from vllm import LLM, SamplingParams

# os.environ["CUDA_VISIBLE_DEVICES"] = "4,5,6,7"
# os.environ["VLLM_USE_V1"] = "1"
# os.environ["TORCHDYNAMO_DISABLE"] = "1"

# Sample prompts.
prompt = "The capital of France is"

# Create a sampling params object.
sampling_params = SamplingParams()


def main():
    llm = LLM(
        model="/root/paddlejob/ERNIE-4.5-Turbo/baidu/paddle_internal/ERNIE-4.5-21B-A3B-PT",
        # model="/root/paddlejob/vllm/Qwen3-30B-A3B",
        dtype="bfloat16",
        tensor_parallel_size=1,
        # enforce_eager=True,
        trust_remote_code=True
        )

    # outputs = llm.generate(prompts, sampling_params)
    
    conversation = [
        {"role": "system", "content": "你是一个乐于助人的助手。"},
        {"role": "user", "content": f"{prompt}"},
    ]
    
    outputs = llm.chat(
        [conversation],
        sampling_params,
        chat_template_kwargs={
        "enable_thinking": False,
        "add_generation_prompt": True, 
        }
    )
    # Print the outputs.
    print("\nGenerated Outputs:\n" + "-" * 60)
    for output in outputs:
        generated_text = output.outputs[0].text
        print(f"Prompt:    {prompt!r}")
        print(f"Output:    {generated_text!r}")
        print("-" * 60)

if __name__ == "__main__":
    main()

1. main branch

issue1: garbled text

baidu/ERNIE-4.5-21B-A3B-PT (tensor_parallel_size=1 or tensor_parallel_size=2)

Capturing CUDA graphs: 100%|█████████████████████████████████████████████████████| 67/67 [00:37<00:00,  1.79it/s]
INFO 07-02 21:24:09 [gpu_model_runner.py:2280] Graph capturing finished in 37 secs, took 3.17 GiB
INFO 07-02 21:24:09 [core.py:172] init engine (profile, create kv cache, warmup model) took 54.31 seconds
WARNING 07-02 21:24:10 [tokenizer.py:262] Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
INFO 07-02 21:24:10 [chat_utils.py:421] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
Adding requests: 100%|███████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1240.92it/s]
Processed prompts: 100%|███| 1/1 [00:00<00:00,  9.36it/s, est. speed input: 187.60 toks/s, output: 150.04 toks/s]

Generated Outputs:
------------------------------------------------------------
Prompt:    'The capital of France is'
Output:    '))));..。</。.}}}) problemen称 oaserJ.}】。)].)],'
------------------------------------------------------------

enforce_eager=True or os.environ["TORCHDYNAMO_DISABLE"] = "1"

Adding requests: 100%|███████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1551.15it/s]
Processed prompts: 100%|█████| 1/1 [00:00<00:00,  1.89it/s, est. speed input: 37.91 toks/s, output: 30.33 toks/s]

Generated Outputs:
------------------------------------------------------------
Prompt:    'The capital of France is'
Output:    'The capital of France is Paris. It is a vibrant and historic city known'
------------------------------------------------------------

Qwen/Qwen3-30B-A3B (only tensor_parallel_size=2 garbled text, but tensor_parallel_size=1 is normal)

Capturing CUDA graphs: 100%|█████████████████████████████████████████████████████| 67/67 [00:47<00:00,  1.41it/s]
Capturing CUDA graphs: 100%|█████████████████████████████████████████████████████| 67/67 [00:47<00:00,  1.41it/s]
(VllmWorker rank=0 pid=39026) INFO 07-02 21:32:30 [custom_all_reduce.py:196] Registering 6499 cuda graph addresses
(VllmWorker rank=1 pid=39029) INFO 07-02 21:32:30 [custom_all_reduce.py:196] Registering 6499 cuda graph addresses
(VllmWorker rank=1 pid=39029) INFO 07-02 21:32:31 [gpu_model_runner.py:2280] Graph capturing finished in 48 secs, took 5.55 GiB
(VllmWorker rank=0 pid=39026) INFO 07-02 21:32:31 [gpu_model_runner.py:2280] Graph capturing finished in 48 secs, took 5.55 GiB
INFO 07-02 21:32:31 [core.py:172] init engine (profile, create kv cache, warmup model) took 76.86 seconds
INFO 07-02 21:32:32 [chat_utils.py:421] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
Adding requests: 100%|███████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1380.61it/s]
Processed prompts: 100%|███| 1/1 [00:00<00:00,  8.00it/s, est. speed input: 240.36 toks/s, output: 128.17 toks/s]

Generated Outputs:
------------------------------------------------------------
Prompt:    'The capital of France is'
Output:    'Lon")). exem newValue deleting gap每一位peated.Fieldimest� ))\n\n⁅𝔱 Jwtfulness'
------------------------------------------------------------

enforce_eager=True or os.environ["TORCHDYNAMO_DISABLE"] = "1"

Adding requests: 100%|███████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1392.07it/s]
Processed prompts: 100%|█████| 1/1 [00:01<00:00,  1.07s/it, est. speed input: 28.13 toks/s, output: 14.07 toks/s]

Generated Outputs:
------------------------------------------------------------
Prompt:    'The capital of France is'
Output:    'The capital of France is **Paris**. 🇫🇷✨'
------------------------------------------------------------

issue2: CUDA error: out of memory

ERNIE-4.5-300B-A47B-PT + quantization="fp8" + tensor_parallel_size=8 + 「enforce_eager=True or os.environ["TORCHDYNAMO_DISABLE"] = "1"」==> run success
ERNIE-4.5-300B-A47B-PT + quantization="fp8" + tensor_parallel_size=8 ==> run failed CUDA error: out of memory

Qwen3-235B-A22B-FP8 + tensor_parallel_size=4 + 「enforce_eager=True or os.environ["TORCHDYNAMO_DISABLE"] = "1"」 ==> run success
Qwen3-235B-A22B-FP8 + tensor_parallel_size=4 ==> run failed CUDA error: out of memory

2. Old version main branch code(May 29 version commit id fd7bb88)

The above problems will not occur, regardless of whether there is 「enforce_eager=True or os.environ["TORCHDYNAMO_DISABLE"] = "1"」， it is normal

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[Isotr0py]

After investigation, here are more information about this issue:

• Seems only unquantized MoE has this problem, running with w8a8 can still generate normal outputs:

python examples/offline_inference/basic/chat.py --model nytopop/Qwen3-30B-A3B.w8a8 -tp 2 --max-tokens 128

Outputs:

Generated Outputs:
--------------------------------------------------------------------------------
Prompt: None

Generated text: '<think>\nOkay, the user asked me to write an essay about the importance of higher education. Let me start by understanding what they need. They might be a student who needs to submit an essay, or maybe someone preparing for a presentation. Either way, the essay should be well-structured and cover key points.\n\nFirst, I need to outline the main arguments. Common points include economic benefits, personal development, societal impact, and career opportunities. I should make sure each paragraph has a clear focus. \n\nI should start with an introduction that sets the context. Maybe mention how higher education is a gateway to opportunities. Then, in the body'
--------------------------------------------------------------------------------
Prompt: None

Generated text: '<think>\nOkay, the user wants an essay on the importance of higher education. Let me start by understanding the key points that should be covered. First, I need to define what higher education is, maybe mention universities, colleges, vocational schools. Then, think about the main arguments.\n\nEconomic benefits come to mind. Higher education often leads to better job opportunities and higher salaries. I should include statistics or studies that show this. Also, how it contributes to the economy as a whole through innovation and skilled workforce.\n\nPersonal development is another angle. Higher education fosters critical thinking, problem-solving skills, and exposure to diverse ideas. Maybe mention'
--------------------------------------------------------------------------------

• MoE without shared_experts like PhiMoE and olmoe can generate normal outputs:

python examples/offline_inference/basic/chat.py --model microsoft/Phi-tiny-MoE-instruct -tp 2 --max-tokens 128

Outputs:

Generated Outputs:
--------------------------------------------------------------------------------
Prompt: None

Generated text: ' Title: The Indispensable Importance of Higher Education\n\nIntroduction:\nHigher education stands as a pivotal pillar of knowledge, growth, and development in the fabric of society. Its importance can be seen from various perspectives, including personal, professional, and societal benefits. This essay will present a detailed analysis on the importance of higher education, touching upon its role in individual success, societal progress, and overall worldwide advancements.\n\nBody paragraph #1:\nFrom a personal standpoint, higher education serves as a catalyst for intellectual growth, critical thinking, and'
--------------------------------------------------------------------------------
Prompt: None

Generated text: " Title: The Importance of Higher Education\n\nThe importance of higher education in today's society cannot be overstated. As the world becomes increasingly complex and competitive, access to quality higher education is more important than ever. Higher education offers numerous benefits not only to individuals but also to society as a whole, including economic growth, innovation, and social mobility.\n\nEducation is the foundation for personal and professional development. It equips individuals with the knowledge, skills, and qualifications necessary to pursue their chosen careers while also promoting lifelong learning. Higher education provides advanced training"
-------------------------------------------------------------------------------

Also cc @bnellnm Would this be related to recent MoE refactoring?

[bnellnm]

I'll take a look.

[zou3519]

@Isotr0py can you get a tlparse? (instructions here, see "quickstart": https://github.com/pytorch/tlparse?tab=readme-ov-file#tlparse-parse-structured-pt2-logs). This gathers all of the compile logs together into some html pages that we can stare at.

[bnellnm]

I was able to run

python examples/offline_inference/basic/chat.py --model Qwen/Qwen3-30B-A3B -tp 2 --max-tokens 128

with latest main w/o problems.

Recently, main was in a state where some FusedMoE ops would give a wrong answer due to the registration of the moe_forward op. You may have hit that version?

[Isotr0py]

with latest main w/o problems.

Recently, main was in a state where some FusedMoE ops would give a wrong answer due to the registration of the moe_forward op. You may have hit that version?

Yes, I just pull from main branch agian and re-run the command for Qwen3 MoE, it can generate common outputs now. Perhaps the registration of the moe_forward op is the cause.

@CSWYF3634076 Can you check if this issue still encountered in ERNIE MoE with latest main branch?

[CSWYF3634076]

with latest main w/o problems.

Recently, main was in a state where some FusedMoE ops would give a wrong answer due to the registration of the moe_forward op. You may have hit that version?

Yes, I just pull from main branch agian and re-run the command for Qwen3 MoE, it can generate common outputs now. Perhaps the registration of the moe_forward op is the cause.

@CSWYF3634076 Can you check if this issue still encountered in ERNIE MoE with latest main branch?

@Isotr0py @bnellnm
I pulled this latest code and the issue1 has been fixed. It's great！
My local code the issue1 commit is aa0dc77 (Jun 27)
However, the memory issue with issue2 still exists. Will the latest CUDA graph use more memory？