Repository navigation
vllm-xpu: default to XPU graphs on vLLM v0.31.0+ instead of forcing eager mode - #34
Merged
Merged
Conversation
…om v0.31.0 From vLLM v0.31.0 the XPU backend captures graphs by default and VLLM_XPU_ENABLE_XPU_GRAPH no longer exists, so --enforce-eager now disables exactly what the version enables. Measured cost is roughly 2x decode (TPOT 64 ms vs 36 ms, 4x Arc Pro B70 at concurrency 8) with TTFT unchanged. - vllm-xpu-run: drop the flag from the quickstart, multi-GPU, and remote-deploy launch lines; rewrite the flag-table row and the CUDA translation row to say when (pre-v0.31.0 only) it is still warranted, with a one-liner to check the image version. - emit_launch.sh / recommend.py: graphs on by default; add an opt-in --enforce-eager flag for older images. - vllm-xpu-bench, vllm-xpu-profile: remove the flag from the example launches; note that an eager baseline understates real throughput and that wide inter-step gaps on v0.31.0+ are a real finding, not noise. - torch-xpu-bench: reframe the compile A/B as deciding whether to add eager back for a problem model, not whether to drop it. - generate_agents.py: read and write with explicit utf-8 so the generator runs on Windows Python (cp1252 default raised on the non-ASCII glyphs in SKILL.md bodies). Signed-off-by: Gopesh Khandelwal <gopesh.khandelwal@intel.com>
GopeshKh
requested review from
258SusanLiu,
YuningQiu and
unrahul
as code owners
October 23, 2026 15:08
recommend.py stopped emitting --enforce-eager, but calibrate.py and verify.py still hardcoded it. Verification measured an eager server against a graphs-on recommendation, and calibration derived BWE from an eager run (0.58 vs 0.75 graphs-on, 1x Arc Pro B70, v0.32.0), skewing every --use-calibration prediction. - verify.py, calibrate.py: launch without --enforce-eager; add an opt-in --enforce-eager flag for pre-v0.31.0 images. - calibrate.py: record the mode in the cache and key eager runs to a separate __eager file so the two modes do not overwrite each other. - recommend.py --use-calibration: only use a calibration taken in the same mode as the emitted launch line. Files without the mode key predate this change and were always eager, so they count as eager. - SKILL.md: note that all three tiers must agree on the mode. - tests: cover default and opt-in launch args for both scripts, the cache key, and calibration mode matching including legacy files. Signed-off-by: Gopesh Khandelwal <Gopesh.Khandelwal@intel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed and why
vLLM v0.31.0 turned XPU graph capture on by default and removed
VLLM_XPU_ENABLE_XPU_GRAPH. The vLLM skills,emit_launch.sh, and themodel-config-recommendlaunch line still hardcoded--enforce-eager, a flagthat now disables exactly what the version enables. The current
vllm/vllm-openai-xpu:latestis v0.32.0, so anyone following the skills today serves at roughly half the decode throughput they should get.This PR makes graphs-on the default everywhere. Eager stays available as an explicit opt-in for pre-v0.31.0 images. The PR also corrects the bench and profile guidance, so users stop benchmarking and profiling a configuration
nobody should deploy.
vllm-xpu-run: drop
--enforce-eagerfrom the quickstart, multi-GPU, and remote-deploy launch lines. Rewrite the flag-table row and the CUDA translation row to say when the flag is still warranted (pre-v0.31.0 images, or isolating a suspected graph-capture bug). Add a one-liner to check the image's vLLM version.emit_launch.sh/recommend.py: graphs on by default. New opt-in--enforce-eagerflag for older images.vllm-xpu-bench / vllm-xpu-profile: remove the flag from the example launches. Note that an eager baseline understates real throughput, and that wide inter-step gaps on v0.31.0+ are a real finding rather than eager-mode
noise.
torch-xpu-bench: reframe the compile A/B as deciding whether to add eager back for a problem model, not whether to drop it.
scripts/generate_agents.py: read and write with explicit UTF-8, so the generator runs on Windows Python. The cp1252 default raisedUnicodeDecodeErroron the non-ASCII glyphs in SKILL.md bodies. Output on Linux is unchanged.Behavior check of the emitters:
recommend.pybehaves the same with and without--enforce-eager.Type of change
Validation
bash scripts/check-skills.shpasses locallypython3 scripts/generate_agents.pywas run after anySKILL.mdfrontmatter change (or the pre-commit hook handled it)
HOW_TO_TEST.mdwas run on real hardware, or the reason it was notis explained above
git commit -s), per the DeveloperCertificate of Origin in CONTRIBUTING.md
Also passing:
tests/static.sh,tests/secure-config.sh(22/22),tests/model-fit.sh.Benchmark data
--enforce-eager(previous default)4x Intel Arc Pro B70, concurrency 8, vLLM-XPU v0.31.0+. Same model, same image, same hardware; the only difference is the flag.