Add multi-dataset model run prediction uploads

[luke-e-schaefer]

Summary — v0.20.0: multi-dataset model runs

Stacked on #467 (update-nuc-sdk-for-new-eval-stuff-pt1) — review that first; this PR's diff is only the 7 files below. The SDK-visible half of the scaleapi change that gives a model run a resolved set of datasets instead of one declared dataset.

The one new method

dataset_b.upload_predictions_for_model_run(run_id, predictions)

Posts to dataset/{dataset_id}/modelRun/{model_run_id}/uploadPredictions, adding dataset_b to the run's dataset set. That is what lets a single run be scored against a benchmark whose items span several datasets — previously create_benchmark_evaluation_v2 returned a 400 for exactly that case.

Same update / asynchronous / batch_size / file-batching / trained_slice_id arguments as upload_predictions, and it runs the same duplicate-id check.

What is deliberately not changed

Dataset.upload_predictions still cannot widen a run, and that's the point. It identifies the run by (dataset, model), so it finds the run already on this dataset or creates a new one. Widening got its own endpoint rather than loosening this one — mirroring the server, where the existing route kept its never-widen contract exactly as it was.

ModelRun.predict also stays on the old route. It's deprecated, and switching it would silently turn a stale dataset_id passed to get_model_run() into a widening upload. Its docstring now says it fails for multi-dataset runs and points at the new method.

Routing

PredictionUploader now accepts dataset_id together with model_run_id — previously an assertion rejected the pair — and picks the endpoint from which identifiers are present:

Arguments	Route	Can widen?
dataset_id + model_run_id	dataset/{ds}/modelRun/{run}/uploadPredictions	Yes
dataset_id + model_id	dataset/{ds}/model/{model}/uploadPredictions	No
model_run_id alone	modelRun/{run}/predict	No (deprecated)

The two pre-existing forms route exactly as before.

Access, and the sharp edge

Widening requires write on this dataset and on every dataset the run already covers. That's stricter than it looks necessary, for a reason worth knowing: a run is visible only to users who can read all of its datasets, so adding a dataset to a run can remove that run from a collaborator's view. The set also only ever grows — a later upload never drops a dataset, so it can't widen who can read the run.

Docstring corrections

Both are wrong as of the server change, not merely incomplete:

• create_benchmark_evaluation_v2 / Benchmark.create_evaluation_v2 said the run's predictions "must cover items from the benchmark's datasets". The server enforced that with a 400; it no longer does. Coverage may be partial or empty, and uncovered members score as false negatives.
• ModelRun.predict — see above.

Tests

tests/test_multi_dataset_model_runs.py — 9 mock-based tests pinning all three routes (the whole difference between them is the route), the async route, trained_slice_id forwarding, and that the new entry point runs the duplicate-id check rather than bypassing it.

Ran locally: 68 passed across the new file plus the benchmark / eval-v2 / preset / leaderboard suites. black and isort clean on the changed files. Diff is purely additive — no reformatting churn from a newer local black.

Server dependency: the new route ships with the multi-dataset model-run work in scaleapi (#154100). Unit tests pass regardless; live calls 404 until that deploys.

🤖 Generated with Claude Code

Greptile Summary

Adds multi-dataset model-run prediction uploads.

• Adds Dataset.upload_predictions_for_model_run, including synchronous and asynchronous upload paths.
• Extends PredictionUploader routing to support dataset and model-run identifiers together.
• Documents relaxed benchmark coverage requirements and the deprecated single-dataset ModelRun.predict behavior.
• Bumps the package version to 0.20.0 and adds route-focused tests.

Confidence Score: 5/5

The PR appears safe to merge, with no concrete changed-code defects identified.

The new synchronous and asynchronous paths preserve the existing prediction payload, batching, duplicate-ID validation, and job-handling behavior while selecting the intended dataset-and-model-run endpoint.

Important Files Changed

Filename	Overview
nucleus/dataset.py	Adds synchronous and asynchronous public entry points for uploading predictions from a dataset to an existing model run.
nucleus/annotation_uploader.py	Adds the dataset-plus-model-run route while preserving the two existing routing forms.
tests/test_multi_dataset_model_runs.py	Covers route selection, asynchronous routing, option forwarding, and duplicate-ID validation.
nucleus/model_run.py	Documents why the deprecated prediction method does not support multi-dataset runs.
nucleus/init.py	Updates benchmark-evaluation documentation to permit partial or empty dataset coverage.
nucleus/benchmark.py	Aligns benchmark evaluation documentation with multi-dataset and partial-coverage behavior.
pyproject.toml	Bumps the client package version from 0.19.0 to 0.20.0.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[PredictionUploader identifiers] --> B{dataset_id and model_run_id?}
    B -->|Yes| C[dataset/dataset_id/modelRun/model_run_id/uploadPredictions]
    B -->|No| D{model_run_id only?}
    D -->|Yes| E[modelRun/model_run_id/predict]
    D -->|No| F[dataset/dataset_id/model/model_id/uploadPredictions]
    C --> G[Run may span multiple datasets]
    E --> H[Deprecated single-dataset inference]
    F --> I[Resolve or create run for dataset and model]

Loading

_{Reviews (1): Last reviewed commit: "Add multi-dataset model run prediction u..." | Re-trigger Greptile}