[JerT33]

What this PR does / why we need it:

Background:
Currently any runtime spec that includes a value of other than node for the trainer replicatedJob name in a Torch, JAX, or XGBoost runtime results in a successful resource creation, but a hanging training job. This PR seeks to be a small, low-risk change that will reject the spec as submission time, instead of failing later during the training process.

Implementation:

• Added validateTrainerReplicatedJobName to the TrainingRuntime and ClusterTrainingRuntime webhooks (create and update), next to the existing check that requires the node container
• Only Torch, JAX, and XGBoost runtimes are checked, since those plugins build the coordinator address from node. MPI and Flux have their own naming rules
• Updates are ratcheted: existing runtimes that already use another name can still be updated
• Added unit and integration tests, and a note in docs/operator-guides/job-template.md

Which issue(s) this PR fixes
Fixes #4422

Live Cluster Testing Evidence

Before:

$ kubectl get pods -l jobset.sigs.k8s.io/jobset-name=renamed-check
NAME                              READY   STATUS    RESTARTS   AGE
renamed-check-trainer-0-0-nzhj9   1/1     Running   0          46s
renamed-check-trainer-0-1-rbd8z   1/1     Running   0          46s

$ kubectl logs renamed-check-trainer-0-1-rbd8z
[E1107 17:19:35.651662840 socket.cpp:1028] [c10d] The client socket has timed out after 60000ms while trying to connect to (renamed-check-node-0-0.renamed-check, 29500).

After (controller image built from this branch at 18a17f1):

$ kubectl get deploy kubeflow-trainer-controller-manager -n kubeflow-system -o jsonpath='{.spec.template.spec.containers[0].image}'
docker.io/library/trainer-controller-manager:pr4423-18a17f1

# 1. A runtime created before this change (trainer replicatedJob named "trainer") can still be updated
$ kubectl get clustertrainingruntime torch-lab-renamed -o jsonpath='{.spec.template.spec.replicatedJobs[*].name}'
trainer
$ kubectl label clustertrainingruntime torch-lab-renamed check=ratchet
clustertrainingruntime.trainer.kubeflow.org/torch-lab-renamed labeled

# 2. Creating the same runtime again is rejected
$ kubectl delete clustertrainingruntime torch-lab-renamed
clustertrainingruntime.trainer.kubeflow.org "torch-lab-renamed" deleted
$ kubectl apply -f runtime-renamed.yaml
Error from server (Forbidden): error when creating "runtime-renamed.yaml": admission webhook "validator.clustertrainingruntime.trainer.kubeflow.org" denied the request: spec.template.spec.replicatedJobs[0].name: Invalid value: "trainer": must be "node" for the replicatedJob labeled trainer.kubeflow.org/trainjob-ancestor-step=trainer in the Torch, JAX, and XGBoost runtimes

# 3. The same runtime with the trainer replicatedJob named "node" is accepted
$ kubectl apply -f runtime-node.yaml
clustertrainingruntime.trainer.kubeflow.org/torch-lab-node created

# 4. Renaming it away from "node" is rejected
$ kubectl patch clustertrainingruntime torch-lab-node --type=json -p '[{"op":"replace","path":"/spec/template/spec/replicatedJobs/0/name","value":"trainer"}]'
Error from server (Forbidden): admission webhook "validator.clustertrainingruntime.trainer.kubeflow.org" denied the request: spec.template.spec.replicatedJobs[0].name: Invalid value: "trainer": must be "node" for the replicatedJob labeled trainer.kubeflow.org/trainjob-ancestor-step=trainer in the Torch, JAX, and XGBoost runtimes

# 5. A TrainJob on the valid runtime still trains to completion
$ kubectl apply -f trainjob-node.yaml
trainjob.trainer.kubeflow.org/node-check created
$ kubectl get trainjob node-check
NAME         STATE      AGE
node-check   Complete   45s
$ kubectl get pods -l jobset.sigs.k8s.io/jobset-name=node-check
NAME                        READY   STATUS      RESTARTS   AGE
node-check-node-0-0-q6c77   0/1     Completed   0          41s
node-check-node-0-1-rk7ft   0/1     Completed   0          41s
$ kubectl logs node-check-node-0-1-rk7ft --tail=2
[rank 1] step 300  loss on my data=1.915
[rank 1] final weights fingerprint:    9a233ae76fdc

Checklist:

• Docs included if any changes are user facing

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign electronic-waste for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment