Disable the torch.compile cache checks when VLLM_DISABLE_COMPILE_CACHE=1

[houseroad]

Disable the checks if it's not used.

Run the benchmark script to test

VLLM_USE_V1=1; with-proxy python3 benchmarks/benchmark_latency.py --model "meta-llama/Meta-Llama-3.1-70B" --tensor-parallel-size 8 --trust-remote-code --enable-chunked-prefill --disable-log-stats --gpu-memory-utilization=0.7

Output

Avg latency: 3.45998974510779 seconds
10% percentile latency: 3.2058258310426027 seconds
25% percentile latency: 3.2742707084398717 seconds
50% percentile latency: 3.4767037329729646 seconds
75% percentile latency: 3.6269146820995957 seconds
90% percentile latency: 3.6901761000510307 seconds

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D72945995

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D72945995

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D72945995

[youkaichao]

LGTM

[zou3519]

I mentioned this to @houseroad offline, but the implication here is:

• if hash_str is None and envs.VLLM_DISABLE_COMPILE_CACHE is True, then this implies that the vLLM monkey-patched hijack_compiled_fx_graph_hash never ran
• if hijack_compiled_fx_graph_hash never ran, but compile_fx produced a graph, then this implies that we are actually relying on torch.compile's caching logic!

The overall inference is that when VLLM_DISABLE_COMPILE_CACHE is on, we can be relying on torch.compile's cache. This doesn't seem right to me (or at least it is confusing).

I think the right fix is something like disabling the remote caches (#16611) so we don't get into this state.

Thoughts, @youkaichao? I'm also happy to document this behavior and ship this PR as-is but I will potentially dig it out later in the standalone_compile refactor.

[youkaichao]

I think the right fix is something like disabling the remote caches (#16611) so we don't get into this state.

agree, let's disable remote cache then.

[houseroad]

Yeah, I agree with the above point, #16611 is the right solution. I think it still makes sense to add this change, my point is that we should have some overall switch to disable all the logic related to torch compile cache, which is VLLM_DISABLE_COMPILE_CACHE.

For some internal use case, with VLLM_DISABLE_COMPILE_CACHE=1, we still fail on the compile cache related checks, which doesn't really make sense to me.