[v0.8.4rc2] FAQ / Feedback | 问题/反馈

[wangxiyuan]

Anything you want to discuss about vllm on ascend.

Please use doc: https://vllm-ascend.readthedocs.io

• Fail to start vllm under /workspace: [Bug]: Fail to start vllm under /workspace #725

  • Workaround:

    • 1. cd ~ or cd /, before you execute the vllm serve or launch openai server
    • 2. **(Update 20250430 9:00 AM) You can ** Use docker pull redownload v0.8.4rc2 image to refresh image

  • Resolved: [Build][Bugfix] Fix source code path to avoid reference error #726

    • we need follow vLLM, should move all source code into /vllm-workspace/, and keep /workspace as workdir:
      [1] https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile#L264C7-L264C23
      [2] https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile#L52

---

请使用 https://vllm-ascend.readthedocs.io 安装

[Pandaxia990]

• 启动服务报错：vllm serve /data/DeepSeek-R1-Distill-Qwen-32B --tensor_parallel_size 2 --served-model-name "QwQ-32B" --max-num-seqs 256 --max-model-len=4096 --host 192.168.41.77 --port 8001
  Traceback (most recent call last):
  File "/usr/local/python3.10/bin/vllm", line 5, in
  from vllm.entrypoints.cli.main import main
  File "/workspace/vllm/vllm/entrypoints/cli/main.py", line 7, in
  import vllm.entrypoints.cli.benchmark.main
  File "/workspace/vllm/vllm/entrypoints/cli/benchmark/main.py", line 4, in
  import vllm.entrypoints.cli.benchmark.serve
  File "/workspace/vllm/vllm/entrypoints/cli/benchmark/serve.py", line 4, in
  from vllm.benchmarks.serve import add_cli_args, main
  ModuleNotFoundError: No module named 'vllm.benchmarks.serve'

[89412825]

我也是这个错

[facerless]

I meet the same error. from vllm.benchmarks.serve import add_cli_args, main
ModuleNotFoundError: No module named 'vllm.benchmarks.serve'

[wangxiyuan]

@Pandaxia990 @89412825 @facerless this is a vllm issue. See: vllm-project/vllm#15812 A soltion by hand is vllm-project/vllm#15812 (comment)

copy vllm_0_8_3/vllm/benchmark folder into /home/tiger/.local/lib/python3.10/site-packages

[tt545571022]

VLLM_USE_V1=1 时是否支持torchair的ge图模式？
VLLM_USE_V1=1 python examples/offline_inference_npu.py 启动脚本，

llm = LLM(model="/dfs/model/DeepSeek-V3-bf16/", max_model_len=1024, tensor_parallel_size=4, trust_remote_code=True, enforce_eager=False, compilation_config= CompilationConfig(level=CompilationLevel.DYNAMO_AS_IS), additional_config={"enable_graph_mode": True})

使用以上命令使能torchair的ge图模式，报错如下：

NotImplementedError: VLLM_USE_V1=1 is not supported with --additional-config.

VLLM_USE_V1=1 时还不支持torchair的ge图模式吗？

[89412825]

@wangxiyuan root@6685f4a26704:/workspace/vllm/vllm/benchmarks 把这个文件夹直接复制过去吗，
root@6685f4a26704:/lib/python3.10/site-packages/benchmarks# ll
total 56
drwxr-xr-x 2 root root 4096 Apr 29 04:21 ./
drwxr-xr-x 4 root root 4096 Apr 29 04:23 ../
-rw-r--r-- 1 root root 5873 Apr 29 04:21 endpoint_request_func.py
-rw-r--r-- 1 root root 36139 Apr 29 04:21 serve.py
-rw-r--r-- 1 root root 2178 Apr 29 04:21 utils.py
root@6685f4a26704:/lib/python3.10/site-packages/benchmarks

复制过去还是报错
Traceback (most recent call last):
File "/usr/local/python3.10/bin/vllm", line 5, in
from vllm.entrypoints.cli.main import main
File "/workspace/vllm/vllm/entrypoints/cli/main.py", line 7, in
import vllm.entrypoints.cli.benchmark.main
File "/workspace/vllm/vllm/entrypoints/cli/benchmark/main.py", line 4, in
import vllm.entrypoints.cli.benchmark.serve
File "/workspace/vllm/vllm/entrypoints/cli/benchmark/serve.py", line 4, in
from vllm.benchmarks.serve import add_cli_args, main
ModuleNotFoundError: No module named 'vllm.benchmarks.serve'

[jyf2100]

使用v0.8.4rc2,报错 ModuleNotFoundError: No module named 'vllm.benchmarks.serve'
使用v0.8.4rc1，可以正常启动调用

[wangxiyuan]

@tt545571022 the error is fixed by vllm-project/vllm#16710 and will be supported in next release. Graph mode will be supported by default in next release as well.

[FrankMinions]

[rank0]: File "/workspace/vllm/vllm/executor/ray_distributed_executor.py", line 117, in _init_executor
[rank0]: self.output_decoder = msgspec.msgpack.Decoder(
[rank0]: TypeError: Type '<function Generator at 0xfffd2e1675b0>' is not supported
How to solve this? My model is DeepSeek-R1-w8a8, using ray backend

[Yikun]

Fail to start vllm under /workspace: #725

[Wenzhiqiang16]

@Yikun 使用v0.8.4rc2版本，推理Qwen2.5-VL-7B-Instruct时，报错[rank0]: self.update_attn_cache(max_seq_len, dtype, device)
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/vllm_ascend/attention/attention.py", line 87, in update_attn_cache
[rank0]: self.attn_mask_cache = self.attn_mask_cache.to(device)
[rank0]: RuntimeError: NPU out of memory. Tried to allocate 60.50 GiB (NPU 0; 60.97 GiB total capacity; 15.63 GiB already allocated; 15.63 GiB current active; 44.87 GiB free; 15.79 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.
[ERROR] 2025-05-06-09:12:57 (PID:10733, Device:0, RankID:-1) ERR99999 UNKNOWN applicaiton exception
使用v0.7.3rc2版本时推理是正常的