feat(cli): expose timing.forward_model (op_level | fpm) on predict and recommend [AIC-1926]

[Arsene12358]

Summary

Adds engine.workers.<role>.timing.forward_model: op_level | fpm to the unified aisimulate predict and aisimulate recommend YAML so the AIC forward-pass model can be selected per worker role. op_level is the default and today's behavior; fpm replays whole-forward (FPM) latency from a collected cell. Until now FPM timing was reachable only through the deprecated compatibility command (aiconfigurator cli estimate --forward-model fpm) or by compiling the YAML from Python and injecting a hand-built timing_model block into the replay spec, and the migration guide listed --forward-model as having no unified-CLI mapping. Linear: AIC-1926.

engine:
  workers:
    decode:
      timing: {type: default, forward_model: fpm}   # op_level is the default

aisimulate predict -c m27.yaml --set engine.workers.aggregated.timing.forward_model=fpm

Design

The runtime already supports FPM: the Rust AicTimingConfig accepts forward_model and passes it to compile_engine. This PR only adds the front end, and no Rust code changes.

• Schema (config/engine.py): TimingConfig.forward_model: Literal["op_level", "fpm"] = "op_level", shared by predict and recommend; fpm is rejected unless timing.type is default.
• Predict path (compiler.py, runner.py): the compiler emits the rank field aic_forward_model only when the value is fpm, so op_level replay specs are byte-identical to before; the runner's AIC alias table folds it into the timing_model.config it already synthesizes and validates the value. Per-worker performance_model_metadata now records forward_model (op_level or fpm) for provenance.
• Recommend path (recommend.py, sweeper/config.py, sweeper/sample.py, sweeper/deploy.py): the per-role value lowers into three pinned SearchSpace fields (agg_forward_model, prefill_forward_model, decode_forward_model, default op_level, validated), travels through unroll_sample, and deploy.py emits the same rank field and metadata. Every emitted candidate YAML spells out timing.forward_model, consistent with the normalization that already materializes every other schema default.
• Fail closed: a candidate whose identity (model, hardware, backend version, parallel shape, quantization) has no FPM cell fails at replay with PerfDataNotAvailableError; the Sweeper records it as a failed candidate (reason category replay_runtime). There is no silent fallback to op_level. A fallback policy is out of scope here (AIC-1770 / refactor(perfmodel): unify estimator construction and CLI policies [AIC-1770] #13). In fpm mode with default capacity, the KV capacity is also capped to the cell's collected decode-KV ceiling (existing Rust behavior, now documented).
• Docs: docs/cli/design.md (field table row + paragraph), docs/cli/migrate-from-aiconfigurator.md (--forward-model now maps to the new field; --database-mode, --transfer-policy, --systems-paths stay unmapped), docs/sweeper/configuration.md.

Coexistence with #13 (AIC-1770)

#13 flattens a canonical ForwardPassPerfModelConfig into the Rust AicTimingConfig via serde(flatten), so the flat forward_model key this PR writes into timing_model.config stays valid before and after it. #13 also adds a sweep-wide forward_model to SearchSpace and builds the estimator config from it in deploy.py; the per-role fields here use different names, so there is no collision, but the merge must thread <role>_forward_model into the per-role estimator construction (otherwise the per-role value would be silently discarded). The intended shape after #13 is per-role fields defaulting to "inherit the sweep-wide value"; that is deliberately not pre-empted here against a draft. Expect textual conflicts in sweeper/config.py, sweeper/deploy.py, their tests and docs/sweeper/configuration.md.

Version-slot note

The bundled FPM cells (h200_sxm/vllm/0.25.1 MiniMax-M2.7, b200_sxm/vllm/0.25.1 GLM-5.2-NVFP4) are collected at backend versions outside the queryable version slots, and the Rust AIC provider enforces the slot gate at engine compile, so FPM replays on them need AIC_ALLOW_UNLISTED_VERSIONS=1. That is pre-existing behavior; the design doc now documents it as a transitional escape hatch (the variable is described in perf_database.py as a transition aid slated for retirement). Follow-up worth its own issue: make FPM cells slot-queryable, or add an allow path to the Rust AicTimingConfig, so the documented FPM path does not depend on it. The tests that replay FPM set the variable explicitly and nowhere else.

Testing

• Unit: schema accept/reject (tests/test_cli_config.py), compiler emission per role and metadata (tests/test_unified_traffic_runtime.py), runner alias, duplicate-configuration conflict and invalid value (tests/test_runner.py), Sweeper fields and deploy payload for agg/disagg/fixed timing (tests/sweeper/test_config.py, tests/sweeper/test_deploy.py), recommend lowering and candidate round trip (tests/test_cli_config.py).
• Wiring proof without any FPM-vs-op_level comparison: the covered M2.7 cell completes under fpm; an uncovered identity (tp2) fails closed under fpm while the same shape replays under op_level (tests/test_unified_traffic_runtime.py).
• End to end: tests/e2e/configs/unified_cli/predict/fpm/01-minimax-m27-h200-tp4-fpm.yaml through python -m aisimulate predict, once as written and once with --set …forward_model=op_level, both completing (kept outside predict/engine/ because that directory's exact case list runs without the env var).
• Acceptance: the YAML path reproduces the previous hand-injected FPM replay results for MiniMax-M2.7 h200_sxm tp4 8k/1k at c1–c256 to a worst relative difference of 5.1e-16 (medians identical), and aisimulate predict --set …forward_model=fpm at c16 gives median TPOT 16.26 ms / median TTFT 445.0 ms, matching the injection-driven run that was compared against InferenceX silicon.
• Full tests suite at head: 511 passed, 5 pre-existing skips; ruff check as in CI: clean.

Behavior changes to be aware of

• performance_model_metadata[<role>]["config"] gains a forward_model key (two goldens updated); under fixed/polynomial timing it records op_level, matching the predict-side schema rule.
• Every aisimulate recommend candidate YAML now carries timing.forward_model explicitly (op_level for existing configurations), like the other schema defaults the candidate normalization already spells out.

Out of scope / follow-ups

Fallback policy for uncovered cells, database_mode / systems_paths / transfer_policy on the unified YAML (AIC-1770, #13), the compatibility command's deprecation message (it fires unconditionally and points at the archived repository's migration guide; separate fix).

🤖 Generated with Claude Code

[linear-code]

AIC-1926 [CLI] Expose forward_model (op_level | fpm) on the unified predict/recommend timing config

Review in Linear

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[jasonqinzhou]

APPROVE · 9.2/10 · medium confidence

What this PR does

This PR exposes per-worker op-level versus whole-forward FPM timing selection in the unified predict and recommend configuration, threads that choice through compilation and sweeper candidate generation, and records it in performance-model provenance.

The change is carefully carried across both predict and recommend paths, and the tests distinguish plumbing from accuracy while explicitly proving that an uncovered FPM identity fails closed instead of falling back silently.

Why this score

The 9.2 score reflects an approval-quality Codex pass with clean targeted tests and CI lint, but the packet recommendation is BLOCKED because the required independent Fable model was not observed. No comments should be posted from this packet; the unverified fallback output is retained only as non-canonical evidence.