You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
TrainerClient.wait_for_job_status polls until timeout for an invalid or empty status on local backends #877
TrainerClient.wait_for_job_status only rejects an invalid status on the Kubernetes backend. On the local process and container backends it is never checked, so a status that can't match just polls until the timeout:
fromkubeflow.trainerimportTrainerClientfromkubeflow.trainer.backends.localprocess.typesimportLocalProcessBackendConfigclient=TrainerClient(backend_config=LocalProcessBackendConfig())
job_name=client.train(...) # any local jobclient.wait_for_job_status(job_name, status={"Completed"}) # Spark spelling, TrainJob uses "Complete"# waits the full 600 s, then TimeoutError
The Kubernetes backend raises ValueError right away for the same call (kubernetes/backend.py, the status.issubset(job_statuses) check). An empty status=set() waits until the timeout on every backend, Kubernetes included, because set() passes the subset check.
I found this while reading the backends side by side. #873 / #874 fix the same thing for SparkClient.
What did you expect to happen?
A ValueError straight away for an empty status or one that isn't a subset of Created, Running, Complete, Failed, as the docstring says ("ValueError: The input values are incorrect"), on every backend.
Environment
Kubeflow SDK from main, Python 3.12. No cluster needed: it reproduces with the local process backend.
Activity