Repository navigation
feat(kv-caching): Let VLLM_DTYPE override float16 - #97
Closed
alexey-temnikov wants to merge 1 commit into
Closed
alexey-temnikov wants to merge 1 commit into
alexey-temnikov wants to merge 1 commit into
Conversation
Keep float16 as the default: on Linux aarch64 vLLM 0.28.0 assumes bf16
support without probing the CPU, so a CPU without BF16 instructions
(FEAT_BF16; Apple M1-class under Docker Desktop or Podman) would crash
on the first prefill. Anyone else can opt in with VLLM_DTYPE=bfloat16;
compose interpolates --dtype ${VLLM_DTYPE:-float16}.
Measured, memory is the same for both dtypes (2-replica peak anon on
Graviton4 8000-8014 MiB bf16 vs 8007-8012 MiB fp16; x86 6886 vs 6879).
Speed:
- Graviton4: notebook 6.4-6.7 s with bfloat16 vs 68-75 s with float16
(decode ~22 vs ~1.4 tok/s; 577-token prefill 0.61 vs 1.56 s).
- x86 (Sapphire Rapids, AMX): cold prefill ~25% faster (1.16 vs
1.54 s), warm about the same, so the warm/cold contrast is weaker;
float16 stays the better default there too.
VLLM_DTYPE=bfloat16 reached vLLM through compose interpolation on
both replicas (dtype=torch.bfloat16 in the logs) and the walkthrough
passed with the canned answers.
README: an optional "CPU dtype" prerequisite with a one-line CPU check.
compose: correct the dtype comment. Docker Desktop and Podman machine
on Apple silicon are hardware-virtualized, so the guest sees the host
CPU's features; the risk is CPUs without FEAT_BF16, not emulation.
Drop "emulated" from the two timeout comments for the same reason.
Signed-off-by: Alexey Temnikov <attemnik@amazon.com>
MatthiasHowellYopp
added a commit
that referenced
this pull request
Oct 1, 2026
…ests Applies the memory and test improvements from Alexey Temnikov's PRs #95, #96, #97, and #98, reconciled onto the uv-based cookbook. Memory (#95): size vLLM's KV pool with --kv-cache-memory-bytes 32M, run the model in the engine-core process (VLLM_ENABLE_V1_MULTIPROCESSING=0), use the V1 CPU runner, decommit freed heap (TCMALLOC_AGGRESSIVE_DECOMMIT), drop the LMCache L1 to 0.05 GiB, and warm each replica up before it reports healthy so the two load peaks don't stack. --no-enable-prefix-caching makes every reuse in the demo come from Valkey rather than vLLM's in-process cache. Two-replica steady state drops from ~11 GiB to about 6.6 GiB; the README asks for 9 GiB (5.5 GiB one replica at a time). Verified locally under a hard 8 GiB cap: no OOM, suite 10/10. Runtime-agnostic tests (#98): the live tests read Valkey over the network with the GLIDE client instead of 'docker exec', so they work on Docker, Podman, or nerdctl, and wait for LMCache's async write-through to settle. Adds the .ci-structural-only marker so CI runs lint and the structural tier without the full stack. One replica at a time (#96) and an optional bf16 dtype override (#97) for the smaller-machine path. Also fixes two issues in the preflight while reconciling: use [^0-9] instead of [!0-9] so pasting the block into an interactive zsh doesn't trip history expansion, and surface the 5.5 GiB one-replica figure in the error message and the smaller-machine section. Co-authored-by: Alexey Temnikov <alexey-temnikov@users.noreply.github.com> Signed-off-by: MatthiasHowellYopp <matthias.howell@improving.com>
Collaborator
|
Applied in 384f76b on cookbook-lmcache-revamp. --dtype ${VLLM_DTYPE:-float16} with the opt-in bf16 note in Prerequisites. I verified the default path (replica logs dtype=torch.float16). I didn't exercise bf16 on this host since its CPU support is uncertain, which is the case the float16 default guards. Closing in favor of that commit. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on
kv-caching/memory-footprint, which is the PR into #76.Summary
This changes
--dtype float16to--dtype ${VLLM_DTYPE:-float16}, so readers whose CPU supports bf16 can opt in withexport VLLM_DTYPE=bfloat16. The default stays float16, because bf16 crashed with SIGILL on a virtualized aarch64 host whose CPU lacked FEAT_BF16 (M1-class).The comment and README also correct a wording error. Docker Desktop and Podman machine on Apple silicon use hardware virtualization, not emulation. The real condition is a CPU without bf16 support.
Verification
Memory is the same with either dtype:
bf16 is faster:
With the variable unset, both replicas log
dtype=torch.float16. With it set, they logdtype=torch.bfloat16. I could not reproduce the M1 SIGILL itself, because no M1 hardware was available.