Skip to content

TrainerClient.wait_for_job_status polls until timeout for an invalid or empty status on local backends #877

Open

Description

@G26karthik

What happened?

TrainerClient.wait_for_job_status only rejects an invalid status on the Kubernetes backend. On the local process and container backends it is never checked, so a status that can't match just polls until the timeout:

from kubeflow.trainer import TrainerClient
from kubeflow.trainer.backends.localprocess.types import LocalProcessBackendConfig

client = TrainerClient(backend_config=LocalProcessBackendConfig())
job_name = client.train(...)  # any local job
client.wait_for_job_status(job_name, status={"Completed"})  # Spark spelling, TrainJob uses "Complete"
# waits the full 600 s, then TimeoutError

The Kubernetes backend raises ValueError right away for the same call (kubernetes/backend.py, the status.issubset(job_statuses) check). An empty status=set() waits until the timeout on every backend, Kubernetes included, because set() passes the subset check.

I found this while reading the backends side by side. #873 / #874 fix the same thing for SparkClient.

What did you expect to happen?

A ValueError straight away for an empty status or one that isn't a subset of Created, Running, Complete, Failed, as the docstring says ("ValueError: The input values are incorrect"), on every backend.

Environment

Kubeflow SDK from main, Python 3.12. No cluster needed: it reproduces with the local process backend.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions