[kaulastudies]

Summary

Fixes dynamic SUT listing for two provider paths that can authoritatively enumerate models/endpoints:

• OpenAI: discovers available models through the existing OpenAI models API and maps them to ModelBench SUT definitions.
• Together dedicated: discovers deployments from the account's Together project endpoints and maps both maker/model and bare deployment names to SUT definitions.

This intentionally does not change:

• Together serverless — its catalog is intentionally treated as too large to enumerate; Add default model lists to dynamic SUT providers that don't auto-discover #1646 is the better place for sample/default-list semantics.
• Indirect SUTs — these are user-mediated and do not have an authoritative remote catalog to enumerate.

Duplicate Together dedicated deployments are de-duplicated by SUT UID, and malformed/empty deployment entries are ignored.

Validation

• OpenAI + Together provider tests: 17 passed, 2 skipped
• tests/modelgauge_tests/test_dynamic_sut_factory.py: passed in isolation
• tests/modelgauge_tests/test_sut_factory.py: passed in a fresh pytest process
• Black: passed
• MyPy on the two source files: passed
• git diff --check: passed

The two regression modules are run in separate pytest processes because test_dynamic_sut_factory.py intentionally defines invalid DynamicDriverSUTFactory subclasses while testing DRIVER_NAME validation; running test_sut_factory.py afterward in the same interpreter can expose those temporary subclasses to factory auto-discovery.

All new provider-listing behavior is covered with mocked, network-free tests.

Closes #1644

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[kaulastudies]

recheck

[kaulastudies]

Refreshed this PR against current main and revalidated the actual changed paths.

Validation on the refreshed branch:

• OpenAI/Together focused regression suite: 12 passed, 2 skipped
• git diff --check: passed
• Black on touched files: passed
• current ModelBench MyPy command: passed (notes only)

I also ran the full suite locally on Windows. Its remaining failures are environment-specific and unrelated to this diff (Linux-only SIGALRM / faulthandler.register, missing Graphviz system binaries, Windows path-regex behavior, the fork remote URL assertion, and a local port-binding collision). Upstream CI runs on Ubuntu, so I am leaving Linux confirmation to the PR checks rather than changing unrelated tests.

Together serverless behavior remains unchanged. Ready for renewed review when convenient.

[kaulastudies]

@mlcommons/ai-safety-engineers Could a maintainer review the current OpenAI/Together dedicated discovery patch, 57fdaf5c, when convenient?

The latest build and CLA checks pass. The scope remains limited to providers that can enumerate their models or dedicated endpoints; Together serverless behavior is unchanged. The refreshed regression and type-check results are recorded above.

If the preferred API or project direction has changed, I can adjust the patch. Thank you.