Skip to content

fix(trainer): default to torch-distributed runtime in LocalProcessBac… - #937

Open
harshgupta257 wants to merge 1 commit into
kubeflow:mainfrom
harshgupta257:fix-local-default-runtime
Open

harshgupta257 wants to merge 1 commit into
kubeflow:mainfrom
harshgupta257:fix-local-default-runtime

Conversation

@harshgupta257

Copy link
Copy Markdown

What this PR does / why we need it

TrainerClient.train() documents that runtime defaults to the torch-distributed runtime, and the Kubernetes and Container backends implement that. LocalProcessBackend.train() raised ValueError: Runtime must be provided for LocalProcessBackend instead, so the local examples in the quickstart and local-development guides (which call client.train(trainer=...) without a runtime) failed.

This resolves the default runtime exactly as the other two backends do:

if runtime is None:
    runtime = self.get_runtime(constants.DEFAULT_TRAINING_RUNTIME)
elif isinstance(runtime, str):
    runtime = self.get_runtime(runtime)

The unit test case that asserted the error now asserts that training succeeds with the default runtime.

Fixes #936

Testing

  • make verify passes.
  • make test-python: 717 passed. The updated test case fails on main without this change.
  • Ran the local-development guide's example end to end: the job is now submitted with runtime torch-distributed instead of raising.

Checklist

  • Unit tests updated
  • make verify and make test-python pass

This change was drafted with AI assistance and reviewed, tested, and verified locally by me, per the Kubeflow AI policy.

…kend

TrainerClient.train() documents that runtime defaults to the
torch-distributed runtime, and the Kubernetes and Container backends do
so. LocalProcessBackend instead raised "Runtime must be provided", so
the local examples in the quickstart and local development guides,
which call client.train(trainer=...) without a runtime, failed.

Resolve the default runtime the same way as the other backends.

Signed-off-by: harshgupta257 <harshg.pro257@gmail.com>
@google-oss-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign astefanutti for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: Ready to Merge

Development

Successfully merging this pull request may close these issues.

bug(trainer): LocalProcessBackend.train() fails when runtime is omitted, contradicting docs and other backends

1 participant