DEP: Production hardening for persistent GMS KV failover #15035

/nix/store/10dxp0qxqxxsyiljrh2kp0xqhz6arhcx-bash-5.3p15/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8): No such file or directory

Summary

Harden persistent GMS KV failover for vLLM and SGLang after their engine-specific MVPs. This train keeps inference-engine scheduling, hashes, cache indexes, and eviction policy native to each engine. GMS owns daemon-lifetime GPU allocations plus generation-fenced recovery metadata.

All implementation PRs remain drafts, stacked on Dynamo main 27f2375f9c8.

Parent engine trackers: #12053 (vLLM) and #14704 (SGLang). Recovery-strategy discussion: #14828.

Architecture

engine-native allocator and prefix index
              |
engine adapter: transactional publish/adopt
              |
generation-fenced content directory
              |
block validity, TP agreement, writer retirement
              |
daemon-owned persistent GPU allocation (v0 or v1)

There is one serving owner. A replacement may reuse exact-generation sealed KV after writer retirement and safe classification. Ambiguous predecessor accesses remain quarantined unless a separately qualified GPU-quiescence provider proves they can be reclaimed. Process death, heartbeats, traffic silence, and fixed delays are not treated as CUDA-quiescence proof.

Prerequisite backend PRs

Hardening review order

Final public tip: 3d9f428d0ba5a6c870f724383039172809b46a8d.

Safety boundary

The earlier vLLM and SGLang checkpoints demonstrate the engine integrations and controlled crash loop. The complete arbitrary-crash safety model is cumulative through this hardening train. Reviewers should not infer final quarantine or writer-retirement guarantees from an earlier engine-only checkpoint.

Required invariants:

• Never serve a directory record whose slot generation is stale.
• Never admit a replacement writer before predecessor writers are retired and predecessor KV is classified.
• Require TP ranks to agree before allocation, adoption, reclamation, or takeover succeeds.
• Treat malformed, missing, reordered, or incompatible recovery metadata as a miss or startup failure.
• Keep directory and TP coordination off the per-token hot path.
• Reclamation is owner-scoped. A successor that dies before phase two leaves stamped quarantine; under the opt-in process-death policy the next successor inherits and reclaims it. A strict GPU proof never inherits.
• Reclaim status is per rank and keyed by engine; a record stays pending until a terminal outcome.
• Preserve engine-native capacity, free-list, refcount, and rollback semantics.

Scope

Included: local persistent HBM, vLLM/SGLang integration, TP recovery coordination, request continuity, v0/v1 pool backends, and failover hardening.

Excluded: KVCR tiering, host/storage tiers, remote transfer, router-directed P2P, decode-to-decode transfer, TensorRT-LLM, node/GPU-reset durability, and authenticated same-user multi-tenancy.

Validation status

Tip fdfb07ebeff.

Review fixes:

• DEP: Persistent GPU allocation lifetime across engine sessions #14832: release can name the expected allocation_id, and the daemon refuses a stale incarnation (feat(gms): define persistent KV allocation protocol #12056, feat(gms): own persistent KV allocations in the daemon #14576). List-then-release cleanup passes the listed ID (feat(gms/vllm): attach KV cache to persistent GMS pools #14581, feat(gms/vllm): activate persistent KV worker integration #14583). A same-session exclusive re-claim is idempotent (feat(gms): own persistent KV allocations in the daemon #14576). A forked child cannot keep a parent's session alive (feat(gms): reconnect persistent KV pools through torch #14577). The pool backends thread the ID (refactor(gms): abstract persistent pool ownership backend #14814); the v1 backend refuses identity-qualified destroy (feat(gms): implement persistent pools for v1 #14818).
• feat(gms/vllm): adopt persistent KV blocks through BlockPool #13398: a failed lease seal on the vLLM free path is lost ownership and stops the engine (GMSKVLeaseFenced). A block listed twice is freed once, as in native BlockPool. The three lease guards that keep live blocks from a second consumer each have a test that fails when the guard is removed. The engine-core hook tests carry vLLM-lane markers.

Checks:

• Lint (pre-commit) and CODEOWNERS strict/drift checks are clean.
• GMS unit suite on every touched branch: no new failures; the tip passes 495. vLLM and SGLang component suites show only pre-existing failures.

Failover qualification, vLLM and SGLang, both SIGKILL and real CUDA illegal-address faults:

• Local TP2 (Qwen3-0.6B): three consecutive failovers per run. Visible stream pause 0.6–1.0 s, exact replay, cached KV reused.
• TP8 (Qwen3-235B-A22B-FP8; 8 single-GPU pods, one node; two consecutive failovers per deployment):
  • 8/8 failovers with 0 request errors, cached KV reused, and 8/8 ranks reclaimed.
  • Max stream gap 1.65–2.57 s; first confirmed shadow request 2.08–3.17 s.
• TP16 (Qwen3-235B-A22B-FP8; 16 pods across two nodes; two consecutive failovers per deployment), per run below.
• vLLM replays were exact.

TP16 results:

"Max stream gap" is the longest pause seen by an in-flight streaming request. "First standby request" is the time from the fault until a new request completes on the promoted engine.

Known limitations and open items:

• The default gpu-proof policy cannot reclaim after a hard crash (MPS returns CUDA 201 or times out), so the predecessor's writable pages stay quarantined until a fresh restart. Large-scale runs used the opt-in best-effort process-death-timeout policy.
• SGLang output cannot be byte-exact across a replay boundary even without GMS; a healthy-engine control diverges in the same token range.
• A GPU fault tears down the whole MPS server, including the GMS weight/KV clients on it; whether those GMS servers can make new allocations afterwards is untested.
• Not yet covered: near-full-HBM pressure at TP16.