test(functional-ci): add a backend capability table and shared case helpers

[knowicki-nvidia]

Summary

Resolves BER-528. Based on #77 (top of the BER-425 tests/functional_ci stack).

Adds one backend table, tests/functional_ci/backends.py, and removes the per-file dicts BACKENDS, SHAPE_BACKENDS, TOPOLOGY_BACKENDS, JIT_BACKENDS, and ONNX_BACKENDS.

BACKENDS: dict[str, BackendCase]  # id -> BackendCase(factory, tolerance, capabilities, artifact_format)

@with_backends("dynamic_batch")   # == pytest.mark.parametrize("case", [cases with all given capabilities], ids=[backend ids])
def test_recorded_batch_range(case: BackendCase, device): ...

• Capabilities: save_load, export, dynamic_batch, structured_io, multigraph, onnx_input, jit. The table builds them with _capabilities(*names: Capability), so pytype rejects a misspelled name.
• export backends (inductor-aot, tensorrt, onnxruntime-cuda) also set artifact_format (pt2, tensorrt_plan, onnx). BackendCase.__post_init__ asserts that both are set or neither is.
• The table records only the coverage that exists now. Later sub-issues (BER-529..532) add capabilities, for example dynamic_dims in BER-529.
• A backend without a capability gets no case. pyproject.toml sets empty_parameter_set_mark = "fail_at_collect", so a capability that no backend has fails collection; pytest does not skip it silently. This option applies to the whole repository. aitune and tests/unit still collect without errors.

Shared helpers in lifecycle.py, used by every functional case (including test_precision, test_strategies, test_kernel_selector):

• pinned_strategy(backend) replaces 6 copies of OneBackendStrategy(b).enable_performance_validation(False) + enable_find_max_batch_size(False).
• Tolerance.assert_close(actual, expected) replaces 11 unpacked torch.testing.assert_close calls. Tolerance is frozen because cases share instances.
• assert_matches_eager(tuned, reference, device, tolerance, batch_sizes=(2,), seeds=(5,)) replaces the eager-comparison loops, including two different _assert_matches_eager helpers.
• selected_backend(module, expected=None) also checks the selected backend type. This replaces 6 type(selected_backend(...)) is ... asserts.
• run_lifecycle returns the restored module; the LifecycleResult dataclass is removed.
• models.seeded(device, seed) replaces the RNG block that test_topology copied from make_mlp.

Changes that are not pure moves:

• New test_exported_artifact[inductor-aot|tensorrt|onnxruntime-cuda] (@with_backends("export")) runs the native artifact. Before, test_backend_lifecycle did this behind an artifact_format is None early return, on the reloaded module. Now it uses the tuned module, so 3 extra toy tunes run.
• test_precision builds TensorRT with tensorrt_backend(**overrides) from backends.py and replays every case on two new inputs (seeds 11 and 12). Before, only cuda-graphs-fp32 did the replay.
• test_structured_io widens atol 5x for every backend (row sums in stats accumulate per-element error) instead of the TensorRT-only TENSORRT_ROW_SUM. TensorRT keeps rtol=1e-3, atol=5e-3; Inductor goes from atol=1e-5 to 5e-5.
• test_partial_wrap_multigraph[inductor-aot] now uses the Inductor tolerance from the table (rtol=1e-4, atol=1e-5). Before, it used TENSORRT_FP32 for both backends.
• Shape tests use the shared default input seed instead of 10 + batch_size; existing-ONNX tests use it instead of 20 + batch_size.
• The TensorRT plan runner asserts contiguous inputs. Before, it passed tensor.contiguous().data_ptr(), which can point to a temporary copy.
• onnxruntime-cuda in the shape and existing-ONNX tests now sets execution_provider=CUDA explicitly. Before, it used None, which defaults to CUDA.

Verification

• pytest --co tests/functional_ci: the 39 test ids of test(functional-ci): cover KernelSelectorBackend attention selection #77 plus the 3 test_exported_artifact ids (diffed).
• pytest --co aitune tests/unit passes with fail_at_collect.
• pre-commit passes. pytype 2024.10.11 on Python 3.11 reports no errors for tests/functional_ci.
• No GPU on the VM. The Functional CI GPU job runs the cases.

Link to Devin session: https://nvidia-cloud.devinenterprise.com/sessions/73f65ef5923744f7902f69e2ce14574a
Open in Devin Desktop: https://nvidia-cloud.devinenterprise.com/desktop/session/73f65ef5923744f7902f69e2ce14574a?variant=devin
Requested by: @knowicki-nvidia

Summary by CodeRabbit

• Tests
  • Centralized functional test coverage across supported backends, including lifecycle, ONNX, JIT, shapes, topology, precision, and strategy behavior.
  • Added checks that exported artifacts run natively and match eager execution within backend-specific tolerances.
  • Expanded output comparisons across batch sizes and seeds, including restored models.
  • Test collection now fails when no backend matches the requested capabilities.

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access that start with 'Devin'.

• Disable automatic comment, CI, and merge conflict monitoring

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: ai-dynamo/aitune/.coderabbit.yml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 4c0f91a5-cff0-45bd-9ea8-ecf89af3bb3d

📥 Commits

Reviewing files that changed from the base of the PR and between fe2fe0d and 243af83.

📒 Files selected for processing: 12

• tests/functional_ci/artifacts.py
• tests/functional_ci/backends.py
• tests/functional_ci/lifecycle.py
• tests/functional_ci/models.py
• tests/functional_ci/test_backend_lifecycle.py
• tests/functional_ci/test_existing_onnx.py
• tests/functional_ci/test_jit.py
• tests/functional_ci/test_kernel_selector.py
• tests/functional_ci/test_precision.py
• tests/functional_ci/test_shapes.py
• tests/functional_ci/test_strategies.py
• tests/functional_ci/test_topology.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 8 remain after this review.

📝 Walkthrough

Functional CI tests now use shared backend cases, capability-based selection, and per-backend tolerances. Lifecycle helpers provide pinned strategies and eager-output comparisons. Export tests run artifacts for backends that declare export capability.

Changes: Functional CI backend selection

Layer / File(s)	Summary
Shared backend cases and selection tests/functional_ci/backends.py	Defines backend capabilities, factories, tolerances, artifact formats, and a decorator that selects backends matching requested capabilities.
Lifecycle helpers and seeded model setup tests/functional_ci/lifecycle.py, tests/functional_ci/models.py	Adds pinned strategies, backend checks, and tolerance-based eager comparisons. Lifecycle execution returns the restored module. Model construction uses a seeded RNG context.
Lifecycle and artifact checks tests/functional_ci/test_backend_lifecycle.py, tests/functional_ci/artifacts.py, pyproject.toml	Runs lifecycle tests for save/load-capable backends and export tests for export-capable backends. The TensorRT artifact runner checks input contiguity. Pytest fails collection when a parameter set is empty.
Shared-helper adoption in functional tests tests/functional_ci/test_existing_onnx.py, tests/functional_ci/test_jit.py, tests/functional_ci/test_kernel_selector.py, tests/functional_ci/test_precision.py, tests/functional_ci/test_shapes.py, tests/functional_ci/test_strategies.py, tests/functional_ci/test_topology.py	Updates functional tests to use shared backend selection, pinned strategies, and shared output comparisons. Scenario-specific checks and input parameters remain.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Refactor

Suggested labels: review-effort/medium

Merge Risk: ⚪ Minimal · up to 243af

This change only reorganizes the functional CI tests around a shared backend table and helpers. No concrete defect was found. The GPU job will exercise the cases.

Pre-merge checks: 5

Check name	Status	Explanation
Docstring Coverage		Docstring coverage is 95.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 40 functions across 12 files.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main changes: adding a functional CI backend capability table and shared test helpers.

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.