feat: add FPM self-service onboarding and validation

[Arsene12358]

What changes

FPM self-service takes a Hugging Face-style model config through profile review, collection, memory resolution, replay coverage and matched serving validation without requiring a registered op-level model class. Users select designated hardware, joint precision combinations and one or more TP/DEP/TEP configurations. Each configuration gets its own profile and collection plan; successive runs can reuse the required GPUs. Derived facts and planning estimates retain sources and unresolved assumptions, and users review, edit and accept each exact profile.

The agent guide provides six stages and proactive investigation of the pinned checkpoint and runtime. onboard checkpoint / onboard resume preserve incomplete research, per-configuration acceptance and artifacts in one revision-checked file. Changed inputs invalidate affected downstream progress while preserving historical evidence. Multimodal checkpoints are accepted for text-decoder timing. Users are not asked to invent activation or other non-KV memory bounds: fresh profiles keep memory pending until observation, while complete explicit legacy resource declarations remain supported.

Collection and memory

onboard collect-fpm --executor kubernetes|slurm integrates the existing common collector and Slurm executor from #158. Slurm uses a caller-owned allocation, an explicit image and mounts, Pyxis/Enroot and a shared campaign path. Exact deployment options persist through preview, execution, explicit resume, checkpoint context and finalization. AISimulate launches benchmark workers; initialized vLLM state and Dynamo self-benchmark determine the feasible measurement grid within reviewed limits. This does not introduce an AgentX-specific collection grid.

New requests propose runtime-selected graphs, 0.90 memory utilization, context capped at the smaller of the model limit and 256,000 tokens, and 8,192 scheduled tokens / 256 sequences unless supplied bounds apply. Explicit capture remains available. Native cached-prefill real seeding and decode KV warm-up are enabled by default, with explicit overrides and native eligibility/fallback evidence retained. Zero-KV prefill has no prefix to seed; native dense/non-EP decode restrictions remain explicit. Formal timing-table schemas and identity tags are unchanged.

For audited vLLM 0.27.0, collection observes rank-local physical cache allocations, padding, aliases and permanent reservations using synchronous scheduling. onboard finalize verifies the immutable campaign, native timings, complete rank evidence and formal pair, then writes a new resolved profile and simulation plan without changing source artifacts. It uses the minimum compatible request-usable cache budget and rejects incompatible runtime, layout or phase graph settings. Full attention, sliding windows and supported convolution state use canonical Rust grouped sizing, shared byte capacity, transient prefill admission and window eviction. The resolved profile requires separate acceptance.

Editable standard validation

onboard validate-collection and onboard validate-serving --action prepare|run|assess make the following checks part of stages 5 and 6:

1. Verify native point validity and formal rows, plus observed effective attention groups, initialized graph settings, runtime configuration and KV initialization. Execution-only observation supports audited vLLM 0.27.0 and 0.28.0; this does not extend memory finalization beyond 0.27.0.
2. Select a bounded subset from the observed grid and repeat it through native explicit-point collection. Preserve each fresh launch, attempt and sample; report standard deviation, fresh-sample CV and CV including the original published sample. Repeated measurements never overwrite the formal table.
3. Withhold complete coordinates from an isolated copy of the measurement table and query the canonical native direct estimator with denied fallback. Preserve envelope anchors and report prefill/decode errors and unsupported queries separately. Sparse tables remain incomplete.
4. Run a small matched serving benchmark with pinned AIPerf and independently assess per-request TTFT, TPOT and aggregate output throughput. Optional forward observations get separate native predictions; missing instrumentation is explicitly unavailable, while failing supplied forward evidence prevents qualification.

The JSON/YAML campaign policy exposes all numerical settings. Defaults are five fresh repeats, at most twelve selected points per phase cell, maximum CV 0.05, at most sixteen holdouts per phase, seed 42, p95 absolute relative error 0.20, zero unsupported holdouts, throughput relative error 0.20, and dispatch tolerance max(5 ms, 1% of required gap). Policies and selections are frozen with assessments. Threshold-only changes may reuse revalidated raw samples in a fresh assessment; changed repeat counts or selection require new measurements. Policy and validation artifacts use the existing single checkpoint and invalidate assessment without discarding accepted collection.

Collection preparation runs CPU checks only; GPU repeats require explicit --execute. Serving preparation freezes an inspectable recipe; the caller starts the ordinary serving worker, and --action run explicitly sends benchmark traffic. Missing, stale, failed or inconsistent mandatory evidence prevents accuracy: qualified_for_evaluated_scope. Ordinary exploratory prediction/recommendation remains available with resolved memory.

Replay and serving scope

onboard validate-fpm retains the existing cold aggregated Weka/AgentX replay and records native direct-FPM query coverage separately from request completion. Missing timing preserves partial evidence and cannot certify the rest of a trace. A complete collection grid or a measured-point lookup alone does not establish accuracy.

Matched serving initially supports one complete single-stream play whose predecessor recorded API times are all zero/absent or whose idle gaps are all zero. Other timing combinations, branches and corpora remain incomplete. The pinned AIPerf revision is 7db2ba37a62aa80c882bc90eaf61cc8073e2387b, installed in a separate environment using the documented command. Preparation freezes its native chat payloads and the target tokenizer/template, constructs an explicit trace with exact token identities and one-token blocks, and runs ordinary strict native prediction on those target lengths. Source request identities, dependencies and timing stay intact. The serving producer verifies exact /tokenize token IDs before replaying frozen payloads through AIPerf's native raw-payload loader.

Assessment rechecks original collection, repeat samples, formal data, resolved-memory provenance, original and matched replay coverage, observed serving ranks/runtime and raw client measurements. Collection's phase-specific prefix-caching protocol is retained but excluded from cross-phase configuration equality; actual serving prefix reuse must independently match replay. Qualification is limited to evaluated coordinates, configuration and play. It does not establish corpus-wide or untested-concurrency accuracy.

Compatibility and limits

• ModelOpt sidecars may explicitly declare kv_cache_quant_algo: "none". The shared parser now treats that as unquantized KV metadata and preserves the existing BF16 KV/FMHA defaults instead of letting NVFP4/FP8 weight inference promote them to FP8. Original checkpoint metadata stays intact; absent/null metadata and required DeepSeek V4/V4.1 native settings retain their previous behavior.
• Grouped execution remains cold aggregated vLLM, PP1/CP1, HBM-only, non-speculative, with prefix_caching: false. Unsupported prefix reuse, offload, disaggregation and scalar capacity overrides fail explicitly.
• Memory conversion remains audited for vLLM 0.27.0. Other versions can collect timing but retain unresolved memory; ordinary prediction/recommendation rejects pending memory. Observed capacity includes all resident components even when timing models the text decoder only.
• Validation uses fresh output directories and preserves raw campaigns, reused samples and prior reports. Old campaigns missing runtime/deployment evidence cannot be retroactively qualified. New repeat launches require the source collector revision.
• Initialized graph configuration is observed; per-point actual graph dispatch is not claimed. Model/image identity is bound to launch provenance rather than cryptographic storage attestation.
• Current collection-plan schema 11 and formal publication schema 7 remain unchanged by the validation addition. Historical evidence is not silently migrated. Saved onboarding plan templates are unchanged in this update.
• No new interpolation algorithm, measured dataset, frozen golden or core replay scheduler is introduced by the validation addition. Third-party attribution and packaged legal notices remain aligned.

Validation

The explicit unquantized-KV regression was reproduced first through the real collector capability API, then verified with the original pinned Inkling config and sidecar unchanged. The synthetic regression reaches profile-based planning and serialized cells for automatic and explicit BF16 KV selection without a registered model or op timing data. All 551 affected SDK/collector tests and both frozen engine parity suites (386 cases) pass; no golden was changed. These checks cover metadata interpretation and planning, not a new GPU collection run.

Affected local suites cover 2,204 collector/onboarding/FPM cases and 131 native profile/grouped workflow cases. All pass with the collector package root on PYTHONPATH; the final policy fixes additionally pass 151 serving/workflow cases. JSON and YAML policies retain finite nonnegative limits, including zero, limits above 100% and JSON scientific notation. Independent task reviews and fix re-reviews cover source execution mismatches, failed/incomplete status, raw-artifact preservation, held-out coordinate leakage, native target-token prefix reuse, producer failures and editable threshold reassessment.

Canonical Ruff/format, Rust format, shell syntax, documentation checks, copyright, strict ownership/regeneration, packaged legal files and whitespace pass. The release wheel builds; eight module imports, 49 changed packaged file hashes and four public CLI help commands pass outside the checkout. A real local target tokenizer, pinned AIPerf materialization and native replay also pass from the installed wheel: 97/163 input tokens, 96 reused prefix tokens, two completed requests and eight interpolated queries with zero unsupported. Two actual pinned AIPerf processes against a local SSE mock server independently verify producer payloads, identities, units and supported timing semantics. These are CPU/integration checks, not new silicon accuracy evidence.

Existing measured-data holdouts reproduce 32 withheld predictions per model with zero unsupported queries: MiniMax/H200 TP4 prefill/decode p95 errors are 20.08%/2.28%; GLM-5.2/GB200 DEP16 errors are 73.61%/2.34%. Both correctly fail the default 20% prefill criterion. Thresholds were not adjusted to make these experiments pass, and the timing-only fixtures do not qualify serving accuracy.

An earlier bounded live Slurm run at 71487eb4ed7633ea759c5942398f9fc848594220 collected 244 prefill and 37 decode rows for Llama-3.1-8B-Instruct, GB300 TP1, BF16 weights/FMHA/KV and vLLM 0.28.0 / Dynamo 1.5.0. Native validation, publication and resume passed; the allocation was released. That run predates the warm-up defaults and this validation workflow. No new GPU run, Inkling qualification or complete TP/DEP/TEP silicon coverage is claimed. Hosted Full CI remains separate from local checks and Fast CI.

The preceding validation feature update at cae514eec4070608f550adebf796eb51ac02d907 has clean independent whole-branch and standards re-reviews with no remaining findings. The whole-branch review independently ran 681 cases; the standards review ran 386 parity and 50 boundary cases. Its final policy re-review passed 151 affected tests and 39 independent input probes; its release wheel was rebuilt and rechecked outside the checkout. Fast CI, codeowners and DCO pass on that exact head. The PR remains draft; Full CI, human approval and a new silicon accuracy run are still separate.

The metadata fix at current head fa76b1c47452e9279734d59e97d3f1ea6d67294d has clean independent task, standards and whole-branch reviews, with no deferred findings. The final reviewer independently ran 621 affected model/SDK/collector/onboarding tests and eight additional consumer-precedence probes; the task reviewer also passed all 386 frozen engine parity cases. Fast CI, codeowners and DCO pass on this exact head. No new GPU or serving accuracy result is claimed.

Tracking

Stacked on #238, with #196 as the guided CLI foundation. Tracks AIC-1964, AIC-1965 and AIC-1991. Changes from #229 and #300 are included; this updates the existing draft PR and does not depend on draft #46. Merged #158 supplies Slurm FPM collection. Broader AgentX serving alignment and silicon qualification remain follow-up work.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}