bug(spark): wait_for_job_status still uses a mutable default set, not covered by #330/#393/#874 #885

What happened?

wait_for_job_status in the Spark module still uses a mutable set literal as its status default, in all three places that define it:

def wait_for_job_status(
    self,
    name: str,
    status: set[SparkJobStatus] = {SparkJobStatus.COMPLETED},  # <- mutable default
    ...

• kubeflow/spark/api/spark_client.py
• kubeflow/spark/backends/base.py
• kubeflow/spark/backends/kubernetes/backend.py

This is the same anti-pattern reported in #329, but #329 and both of its open fix PRs (#330, #393) only touch the Trainer and Optimizer modules (optimizer/api/optimizer_client.py, optimizer/backends/*, trainer/api/trainer_client.py, trainer/backends/*) — the Spark module isn't in either diff. #874 ("validate requested job statuses") also edits these same three Spark files but only adds validation inside the function body; it doesn't touch the default-argument line either, so the mutable default survives that change too.

Minimal repro, showing the default set object is created once and shared across every call that doesn't pass status explicitly:

import inspect
from kubeflow.spark.backends.kubernetes.backend import KubernetesBackend
from kubeflow.spark.types.types import SparkJobStatus

default = inspect.signature(KubernetesBackend.wait_for_job_status).parameters["status"].default
print(default)              # {<SparkJobStatus.COMPLETED: 'Completed'>}
default.add(SparkJobStatus.FAILED)
default2 = inspect.signature(KubernetesBackend.wait_for_job_status).parameters["status"].default
print(default2)             # {<SparkJobStatus.COMPLETED: 'Completed'>, <SparkJobStatus.FAILED: 'Failed'>}

Once any code mutates the shared default set in place (e.g. a caller builds on the default with status.add(...) instead of copying it first), every subsequent call to that method made without an explicit status argument silently treats the mutated set as its target state(s), for the lifetime of the process.

What did you expect to happen?

Following the same fix already agreed for #329: default to None and assign a fresh {SparkJobStatus.COMPLETED} inside the function body, so each call gets an independent set, in all three Spark signatures.

Environment

Kubeflow Python SDK version: main at ee22b33.