[lmcafee-nvidia]

What does this PR do?

October 20 main refresh: head d25c8e573af8b72b4333cedfcfe55ac29112323d, parent 07b662d2de468850c62e837c446f70ccf7ec3577, pinned NVIDIA main 474b1e9ee72fbe19205a6dfe26d56620977a6275. Existing branches and dependencies are preserved. The real NIXL handoff test now uses StateHandoffDynamicInferenceEngine and configures its transport through the current setup API.

Initialize newly acquired private MHA value-cache pages before their first model write during admission, resumed decode growth, asynchronous preparation, MTP lookahead allocation, and handoff continuation. This prevents unused V tails from contaminating attention results while preserving cached, shared, imported, and unrelated pages.

Example failure scenario(s)

Scenario (Reusing value pages under cache pressure)

• Requests A through H have uneven prompts and generate 64 tokens each.
• A 10-block pool and a 3-block paused budget force pauses and evictions.
• A resumed request acquires a page whose unused V tail contains NaNs.
• Attention reads that tail and produces nonfinite scores instead of the tokens and scores produced with a roomy pool.
• Initializing the acquired V page preserves finite tails and the exact output comparison.

How was this bug discovered?

The frozen saturation regression poisons unowned value pages and compares generated tokens, prompt scores, and generated scores exactly between roomy and saturated pools, with positive pause and eviction witnesses. Sentinel regressions check that only newly private value pages are cleared. The real NIXL regression separately checks transferred bytes, shared model weights, exact logits and greedy tokens, native transfer completion, and final page ownership.

The current-main audit also found a new MTP lookahead allocator that acquires private pages. Its ownership regression poisons the actual spare page and verifies that the new value plane is cleared, extending the same finite-tail invariant to that path.

Dependency and scope

This PR remains stacked on bi-pairwise-tests at 9b86116173ef6aa225af1ee0fe01af09dec06cc9. Its existing base was refreshed from upstream 4729b9c6cc17af2a1b47bf7c42b05059f36ca3b9 before propagation into descendants. No new peer-campaign dependency was introduced.

Initialize private value pages at admission, resumed decode growth, asynchronous preparation, MTP lookahead allocation, and continuation-only handoff. Preserve byte transfer, shared weights, exact logits/tokens, transfer completion and page-ownership checks. The MTP ownership regression poisons the actual spare page and asserts that its newly allocated value plane is zero.

Imported pages rely on a corrected sender’s finite-tail invariant; older uninitialized senders are not repaired. Historical allocation-heavy measurements showed about 0.85–0.99 ms additional TTFT with equal peak allocated/reserved memory. The October refresh adds no new performance claim.

Validation

October 20 refresh acceptance

Candidate d25c8e573af8b72b4333cedfcfe55ac29112323d includes pinned main 474b1e9ee72fbe19205a6dfe26d56620977a6275.

DFW jobs 20558027: all 5 required groups passed on every required rank, including setup, execution and teardown.

No selected case was skipped.

Some receipts use earlier signed heads b857a9c9008a60020b1a70ee2f66dca85c0ab678. Their production is unchanged; the source audit limits the differences to equivalent fixture setup or test paths not exercised by those receipts. Affected async, Mamba, score-guard, coordinator and transfer witnesses were rerun separately. These receipts are scoped source-equivalence evidence, not executions of the current head.

Validation uses qualified compatibility environments: DFW H100 with Torch 2.10/CUDA 13.1, TE 2.20 and FlashInfer 0.6.16.post3; where listed, HSG GB200 with Torch 2.11/CUDA 13.2, TE 2.20 and FlashInfer 0.6.8. This does not claim the exact current 26.09 CI image.

Signed append-only ancestry, Python compilation, whitespace, Black, isort, Ruff, Pylint and the actual-parent kernel gate passed. Existing documented non-numeric kernel-gate exemptions remain on #65 and #102. No unit tests ran on the local workstation.

Earlier validation records

October 16 handoff CI update: 4471dd50f93555c203f8c626437855d8db5a77c6. The H100 recipe now runs the real NIXL handoff module in a dedicated two-GPU, two-process job for dev and lts; the existing selection script excludes it from both broader eight-process buckets. DFW job 20345348 passed all three cases on both ranks for each selector (12 rank-case observations), with zero failures or skips, normal completion, and coverage export. The rendered recipe and standard CI runner were exercised using the documented Torch 2.10 / CUDA 13.1 / TE 2.20 compatibility runtime; dev/lts select markers in that same runtime, not the exact CI images. Recipe expansion, broad-bucket exclusion, shell syntax, and whitespace checks passed. This commit changes only the H100 unit-test recipe.

October 13 reviewer-action head: b6d5f541a451ccc4e5f0a38bcb9206325b912ffd. Inherits the dedicated FP8 CI jobs from fork/79. Production code and unit-test files are byte-identical to the October 5 head below; no new production behavior is introduced by this merge. DFW job 20129888 passed 10 case executions (foundation dev/lts selection plus the combined descendant tree); HSG job 7923190 passed both MXFP8 executions on GB200. Every dedicated launch used one rank, completed normally, wrote coverage data, and had zero failures or skips. This is focused CI-runner validation, not a full campaign rerun. These runs use the documented DFW Torch 2.10/CUDA 13.1 and HSG Torch 2.11/CUDA 13.2 compatibility runtimes with TE 2.20, not the exact CI images. H100 dev/lts labels exercise selection logic in the same runtime. The tested runner, recipes, production code and tests are byte-identical to the final candidates; the later GitHub action override that forces full Testmon mode was validated statically. Normal CI still needs to exercise its own images and existing maintainer-gated GB200 dispatch.

October 5 candidate: 0dd4f9611845878778dfd035818c8ce47cd112d6. DFW jobs 19787291, 19788491, 19788918 passed 15 selected profile cases (21 rank-case observations) with zero skips and normal fixture teardown. Tested sources: fba7653a01ba4694ea7351bbc2fc0de2be07161a. These receipts precede the inherited one-line media-witness adaptation; all production and selected tests are byte-identical, and final foundation/union media checks qualify that changed helper. HSG job 7685551 adds 6 FA4 profile cases. Inverse job 19789605 reproduced the original failure with normal teardown. DFW uses the GPU-qualified TE 2.20 / FlashInfer 0.6.16.post3 / Mamba 2.3.2.post1 closure with Torch 2.10 / CUDA 13.1; HSG uses TE 2.20 / FA4 with Torch 2.11 / CUDA 13.2. Both are documented compatibility ABI profiles, not the exact 26.08 lock image. Corrected external multiprocessing accounting and a separate current-lock RL import overlay qualify affected DFW commands; original failed allocation receipts remain archived. Syntax, formatting, whitespace and the kernel-determinism coverage gate pass against the actual refreshed parent. The complete twelve-head union remains within the approved 3,050 net unit-test line cap.

[lmcafee-nvidia]

hi @wdykas, following up on this one as keshav suggested—can you please review the private value-page initialization fix? thanks!

[wdykas]

LGTM

[wdykas]

I think the unit test may fail on H100 but that should be an easy fix

[lmcafee-nvidia]

@wdykas fixed in 4471dd5: the handoff tests now have a dedicated two-gpu h100 job and are excluded from the eight-rank jobs. all three cases passed on both ranks for both dev/lts selections on dfw using our compatibility runtime.