Skip to content

vllm-xpu: default to XPU graphs on vLLM v0.31.0+ instead of forcing eager mode - #34

Merged
YuningQiu merged 2 commits into
mainfrom
gk/enable-xpu-graph
Oct 26, 2026
Merged

YuningQiu merged 2 commits into
mainfrom
gk/enable-xpu-graph

Conversation

@GopeshKh

@GopeshKh GopeshKh commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

What changed and why

vLLM v0.31.0 turned XPU graph capture on by default and removed VLLM_XPU_ENABLE_XPU_GRAPH. The vLLM skills, emit_launch.sh, and the model-config-recommend launch line still hardcoded --enforce-eager, a flag
that now disables exactly what the version enables. The current vllm/vllm-openai-xpu:latest is v0.32.0, so anyone following the skills today serves at roughly half the decode throughput they should get.

This PR makes graphs-on the default everywhere. Eager stays available as an explicit opt-in for pre-v0.31.0 images. The PR also corrects the bench and profile guidance, so users stop benchmarking and profiling a configuration
nobody should deploy.

  • vllm-xpu-run: drop --enforce-eager from the quickstart, multi-GPU, and remote-deploy launch lines. Rewrite the flag-table row and the CUDA translation row to say when the flag is still warranted (pre-v0.31.0 images, or isolating a suspected graph-capture bug). Add a one-liner to check the image's vLLM version.

  • emit_launch.sh / recommend.py: graphs on by default. New opt-in --enforce-eager flag for older images.

  • vllm-xpu-bench / vllm-xpu-profile: remove the flag from the example launches. Note that an eager baseline understates real throughput, and that wide inter-step gaps on v0.31.0+ are a real finding rather than eager-mode
    noise.

  • torch-xpu-bench: reframe the compile A/B as deciding whether to add eager back for a problem model, not whether to drop it.

  • scripts/generate_agents.py: read and write with explicit UTF-8, so the generator runs on Windows Python. The cp1252 default raised UnicodeDecodeError on the non-ASCII glyphs in SKILL.md bodies. Output on Linux is unchanged.

Behavior check of the emitters:

$ emit_launch.sh --model Qwen/Qwen2.5-1.5B-Instruct --quant bf16
        --dtype bfloat16 \
        --block-size=64 \
        --max-model-len 4096 \
        --gpu-memory-utilization 0.85

$ emit_launch.sh --model Qwen/Qwen2.5-1.5B-Instruct --quant bf16 --enforce-eager
        --dtype bfloat16 \
        --enforce-eager \
        --block-size=64 \
        ...

recommend.py behaves the same with and without --enforce-eager.

Type of change

  • New skill
  • Fix to an existing skill
  • Performance or accuracy improvement (include benchmark data below)
  • Docs or tooling

Validation

  • bash scripts/check-skills.sh passes locally
  • python3 scripts/generate_agents.py was run after any SKILL.md
    frontmatter change (or the pre-commit hook handled it)
  • For skill behavior changes, the relevant acceptance layer in
    HOW_TO_TEST.md was run on real hardware, or the reason it was not
    is explained above
  • Every commit is signed off (git commit -s), per the Developer
    Certificate of Origin in CONTRIBUTING.md

Also passing: tests/static.sh, tests/secure-config.sh (22/22),
tests/model-fit.sh.

Benchmark data

Config TPOT TTFT
--enforce-eager (previous default) 64 ms unchanged

4x Intel Arc Pro B70, concurrency 8, vLLM-XPU v0.31.0+. Same model, same image, same hardware; the only difference is the flag.

…om v0.31.0

From vLLM v0.31.0 the XPU backend captures graphs by default and
VLLM_XPU_ENABLE_XPU_GRAPH no longer exists, so --enforce-eager now
disables exactly what the version enables. Measured cost is roughly 2x
decode (TPOT 64 ms vs 36 ms, 4x Arc Pro B70 at concurrency 8) with TTFT
unchanged.

- vllm-xpu-run: drop the flag from the quickstart, multi-GPU, and
  remote-deploy launch lines; rewrite the flag-table row and the CUDA
  translation row to say when (pre-v0.31.0 only) it is still warranted,
  with a one-liner to check the image version.
- emit_launch.sh / recommend.py: graphs on by default; add an opt-in
  --enforce-eager flag for older images.
- vllm-xpu-bench, vllm-xpu-profile: remove the flag from the example
  launches; note that an eager baseline understates real throughput and
  that wide inter-step gaps on v0.31.0+ are a real finding, not noise.
- torch-xpu-bench: reframe the compile A/B as deciding whether to add
  eager back for a problem model, not whether to drop it.
- generate_agents.py: read and write with explicit utf-8 so the
  generator runs on Windows Python (cp1252 default raised on the
  non-ASCII glyphs in SKILL.md bodies).

Signed-off-by: Gopesh Khandelwal <gopesh.khandelwal@intel.com>
recommend.py stopped emitting --enforce-eager, but calibrate.py and
verify.py still hardcoded it. Verification measured an eager server
against a graphs-on recommendation, and calibration derived BWE from an
eager run (0.58 vs 0.75 graphs-on, 1x Arc Pro B70, v0.32.0), skewing
every --use-calibration prediction.

- verify.py, calibrate.py: launch without --enforce-eager; add an
  opt-in --enforce-eager flag for pre-v0.31.0 images.
- calibrate.py: record the mode in the cache and key eager runs to a
  separate __eager file so the two modes do not overwrite each other.
- recommend.py --use-calibration: only use a calibration taken in the
  same mode as the emitted launch line. Files without the mode key
  predate this change and were always eager, so they count as eager.
- SKILL.md: note that all three tiers must agree on the mode.
- tests: cover default and opt-in launch args for both scripts, the
  cache key, and calibration mode matching including legacy files.

Signed-off-by: Gopesh Khandelwal <Gopesh.Khandelwal@intel.com>

@YuningQiu YuningQiu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks!

@YuningQiu
YuningQiu merged commit 6e7bce8 into main Oct 26, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants