Repository navigation
docs(kv-caching): Run every step one replica at a time - #96
Closed
alexey-temnikov wants to merge 1 commit into
Closed
alexey-temnikov wants to merge 1 commit into
alexey-temnikov wants to merge 1 commit into
Conversation
Keeping both vLLM replicas resident is what sets the walkthrough's memory floor. Running them one after the other keeps every teaching point, including cross-replica reuse: Valkey and the LMCache server stay up, so replica B still reads the KV that replica A wrote, which also shows the cache outliving the replica that computed it. README: "Running on a smaller machine" now recommends a one-replica- at-a-time flow (start vllm-a; at the cross-replica cell run `compose stop vllm-a` and `compose up -d --wait --no-deps vllm-b`), with a single replica kept as the lighter alternative, and a Podman note. The memory table gains a one-replica-at-a-time column. Notebook: the cold/warm timing section, which uses replica A, moves ahead of the cross-replica section so every replica-A cell runs before the swap; the small-machine note now describes both modes. No cell calls a container CLI, and the default two-replica run is unchanged top to bottom. Measured on x86 (no-swap, memory-limited Docker 29.8 box) with the 32 MiB KV pool: minimum reliable allocation 3.75 GiB (3/3, OOM at 3.5 GiB) versus 6.75 GiB with both replicas resident; with the 128 MiB pool 4.0 GiB on docker (8/8), podman-compose 1.6 (5/5) and podman + docker-compose (1/1). The LMCache server drops the stopped replica's registration after about two minutes with a warning and serves replica B from Valkey without errors. arm64 was not measured in this mode; the README keeps the ~5 GiB guidance. Signed-off-by: Alexey Temnikov <attemnik@amazon.com>
MatthiasHowellYopp
added a commit
that referenced
this pull request
Oct 1, 2026
…ests Applies the memory and test improvements from Alexey Temnikov's PRs #95, #96, #97, and #98, reconciled onto the uv-based cookbook. Memory (#95): size vLLM's KV pool with --kv-cache-memory-bytes 32M, run the model in the engine-core process (VLLM_ENABLE_V1_MULTIPROCESSING=0), use the V1 CPU runner, decommit freed heap (TCMALLOC_AGGRESSIVE_DECOMMIT), drop the LMCache L1 to 0.05 GiB, and warm each replica up before it reports healthy so the two load peaks don't stack. --no-enable-prefix-caching makes every reuse in the demo come from Valkey rather than vLLM's in-process cache. Two-replica steady state drops from ~11 GiB to about 6.6 GiB; the README asks for 9 GiB (5.5 GiB one replica at a time). Verified locally under a hard 8 GiB cap: no OOM, suite 10/10. Runtime-agnostic tests (#98): the live tests read Valkey over the network with the GLIDE client instead of 'docker exec', so they work on Docker, Podman, or nerdctl, and wait for LMCache's async write-through to settle. Adds the .ci-structural-only marker so CI runs lint and the structural tier without the full stack. One replica at a time (#96) and an optional bf16 dtype override (#97) for the smaller-machine path. Also fixes two issues in the preflight while reconciling: use [^0-9] instead of [!0-9] so pasting the block into an interactive zsh doesn't trip history expansion, and surface the 5.5 GiB one-replica figure in the error message and the smaller-machine section. Co-authored-by: Alexey Temnikov <alexey-temnikov@users.noreply.github.com> Signed-off-by: MatthiasHowellYopp <matthias.howell@improving.com>
Collaborator
|
Applied in 384f76b on cookbook-lmcache-revamp. The one-replica-at-a-time flow and the timing-before-cross-replica reorder are in, and the README now carries about 5.5 GiB for that path. Closing in favor of that commit. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on
kv-caching/memory-footprint, which is the PR into #76. Retarget this tocookbook-lmcache-revamponce that one merges.Summary
This adds a low-memory way to run the whole walkthrough, cross-replica reuse included. Only one vLLM replica runs at a time:
docker compose stop vllm-aand thendocker compose up -d --wait --no-deps vllm-b.The notebook's timing section now comes before the cross-replica section, so the swap happens at one point. The default two-replica flow is unchanged. Single-replica mode is still documented as an option.
Verification
Measured as in the parent PR (hard memory limit, no swap):