feat: add Trainer control-plane metrics

cmd/trainer-controller-manager/main.go

@@ -179,8 +179,8 @@ func setupManagerComponents(mgr ctrl.Manager, runtimes map[string]runtime.Runtim
 		os.Exit(1)
 	}
 
-	if err := metrics.SetupServer(mgr, &cfg.Metrics, cfg.TLS); err != nil {
-		setupLog.Error(err, "Could not create metrics server")
+	if err := metrics.Setup(mgr, &cfg.Metrics, cfg.TLS); err != nil {
+		setupLog.Error(err, "Could not create metrics server and instrumentation")
 		os.Exit(1)
 	}
 

docs/operator-guides/index.md

@@ -90,6 +90,20 @@ Integrate with Kueue, Slurm Bridge, KAI Scheduler, Coscheduling, and Volcano
 
 :::::
 
+## Observability
+
+:::::{grid} 1 1 2 2
+:gutter: 3
+
+::::{grid-item-card} Controller Metrics
+:link: metrics
+:link-type: doc
+
+Prometheus metrics for runtime inventory, TrainJob activity, and requested workload capacity
+::::
+
+:::::
+
 ----
 
 ```{toctree}
@@ -105,4 +119,6 @@ runtime-patches
 dynamic-resource-allocation
 extension-framework
 job-scheduling/index
+metrics
+metrics-architecture
 ```

docs/operator-guides/metrics-architecture.md

@@ -0,0 +1,189 @@
+# Trainer metrics architecture
+
+This document summarizes how Trainer metrics are implemented and the design
+decisions behind them.
+
+## Scope
+
+The metric names, labels, and query examples are defined in
+[Trainer Controller Metrics](metrics.md). This document focuses on lifecycle,
+ownership, and implementation decisions.
+
+The proposed accelerator metric is not included yet. Accelerator requests can
+come from runtime defaults, plugins, or TrainJob overrides. We should add this
+metric once the effective-resource rules are clearly defined.
+
+## Architecture
+
+```text
+Kubernetes API
+     |
+     +--> Runtime informers --> Runtime snapshot --> runtime_info
+     |
+     +--> TrainJob informer --> Elected leader --> TrainJob metrics
+                                                     |
+                                             Prometheus registry
+                                                     |
+                                               /metrics endpoint
+```
+
+The implementation uses informer data already held by the controller. Metric
+collection does not call the Kubernetes API. This prevents a slow or
+unavailable API server from blocking a scrape, although collection still
+copies and iterates the local snapshot.
+
+The runtime snapshot and TrainJob event metrics run only on the elected leader.
+The metrics server remains available on every replica, but Trainer metrics are
+owned and exposed by the leader to avoid duplicate series.
+
+### Replica ownership
+
+```text
+Controller replica A                         Controller replica B
+--------------------                         --------------------
+Runtime snapshot: no                         Runtime snapshot: yes
+TrainJob observer: no                        TrainJob observer: yes
+                                             (elected leader)
+
+                         Leader changes
+                A becomes leader <----------> B becomes leader
+                B removes observer            A removes observer
+                A registers observer          B registers observer
+```
+
+## How metrics are recorded
+
+1. Runtime informers watch TrainingRuntime and ClusterTrainingRuntime objects.
+2. Add, update, and delete events maintain the in-memory runtime snapshot.
+3. The runtime snapshot is synchronized before it is used for TrainJob metrics.
+4. The elected leader registers the TrainJob observer.
+5. New TrainJobs increment the creation counter.
+6. The first terminal transition increments either the complete or failed
+   counter and records the lifetime. Repeated terminal updates are ignored.
+
+### TrainJob event flow
+
+```text
+TrainJob created
+      |
+      +--> creation counter
+      |
+      +--> resolve runtime and requested nodes
+             |
+             +--> requested-nodes histogram, when available
+
+TrainJob status update
+      |
+      +--> terminal for the first time?
+             |
+             +-- no --> ignore
+             |
+             +-- yes -> complete/failed counter + lifetime histogram
+```
+
+TrainJobs already present during the initial cache listing are not counted as
+newly created. These counters describe events observed by the current
+controller process; they are not durable historical totals.
+
+## Requested training nodes
+
+The requested-node value is resolved in this order:
+
+1. Explicit TrainJob `numNodes`, including zero.
+2. Runtime ML policy `numNodes`.
+3. The trainer replicated-job template `parallelism` when the runtime has no
+   ML policy.
+4. One node when no runtime value is configured.
+
+If the runtime cannot be found or its value is invalid, the workload-size
+observation is skipped instead of recording a guessed value.
+
+## Labels and cardinality
+
+Labels contain normalized values and bounded categories needed for
+aggregation:
+
+- runtime reference;
+- namespace;
+- runtime kind;
+- result for lifetime observations;
+- normalized failure reason;
+- ML policy; and
+- PodGroupPolicy.
+
+TrainJob names, UIDs, child-job names, and raw error messages are not exported.
+This avoids creating a new time series for every individual job. Runtime and
+namespace labels are still user- or administrator-controlled dimensions, so
+the total number of series grows with the number of configured runtimes.
+
+The lifetime histogram covers ten minutes through 28 days. The
+requested-node histogram covers one through 128 nodes, plus `+Inf`.
+
+## Questions these metrics can answer
+
+These metrics help answer:
+
+- Are users creating TrainJobs, and is the creation rate increasing or
+  decreasing?
+- Which runtimes and namespaces are receiving TrainJobs?
+- Which runtimes are most frequently used?
+- How many TrainJobs finish successfully or unsuccessfully?
+- What is the failure rate by runtime or failure reason?
+- What failure reasons are most common?
+- How long do successful and failed TrainJobs take to reach a terminal state?
+- What are the typical and high-percentile workload sizes by runtime?
+- Which runtimes, ML policies, and PodGroupPolicies are currently configured?
+- Is the Trainer metrics endpoint being scraped successfully?
+
+Use `rate()` or `increase()` for creation and completion activity, and use
+`histogram_quantile()` for lifetime and workload-size percentiles.
+
+Namespace is only a proxy for internal users; these metrics do not identify
+individual customers. They also do not show currently running jobs, resource
+utilization, scheduling wait time, checkpoints, training progress, or model
+quality. Those limitations are described in [Out of scope](#out-of-scope).
+
+### From metric to question
+
+```text
+Metric samples
+      |
+      +--> rate() / increase() ------> creation and completion trends
+      |
+      +--> histogram_quantile() ------> duration and workload percentiles
+      |
+      +--> group by runtime/reason --> usage and failure comparisons
+      |
+      +--> runtime_info --------------> current runtime-policy inventory
+```
+
+## Restarts and leadership changes
+
+Counters and histograms are held in the controller process. Their in-memory
+values reset when the controller restarts or leadership changes. Prometheus
+queries should therefore use `rate()` or `increase()` instead of treating
+them as permanent totals. A follower's metrics endpoint may remain healthy
+while the leader-owned Trainer metric families are absent.
+
+The runtime inventory is rebuilt from informer state after a restart. Events
+that occur before the new leader is ready, or during a leadership transition,
+may not be counted.
+
+Metric collection remains local and does not make Kubernetes API calls. If
+runtime metadata is not available, only the dependent workload-size
+observation is skipped.
+
+## Out of scope
+
+This implementation does not provide:
+
+- durable history across controller restarts;
+- GPU utilization or actual resource consumption;
+- Kueue wait times or scheduling reasons;
+- checkpoint usage;
+- training progress or estimated completion time;
+- hyperparameters or model-quality metrics; or
+- reliable customer identity.
+
+These require additional instrumentation in Trainer, Kueue, JobSet, the
+runtime, or the SDK.

docs/operator-guides/metrics.md

@@ -0,0 +1,93 @@
+# Trainer Controller Metrics
+
+Kubeflow Trainer exposes controller-level Prometheus metrics for platform
+administrators. The metrics describe installed runtimes, TrainJob activity,
+and the workload capacity requested by TrainJobs.
+
+Metrics are exposed by the controller metrics endpoint. TLS, authentication,
+ServiceMonitor resources, and scrape configuration are provided by the platform
+deployment; the controller does not create a Prometheus scrape target by
+itself.
+
+## Metrics
+
+| Metric | Type | Labels | Meaning |
+| --- | --- | --- | --- |
+| `kubeflow_trainer_runtime_info` | Gauge, value `1` | `namespace`, `name`, `api_group`, `kind`, `ml_policy`, `pod_group_policy` | Current `TrainingRuntime` and `ClusterTrainingRuntime` inventory. The namespace is empty for cluster-scoped runtimes. |
+| `kubeflow_trainer_trainjob_created_total` | Counter | `namespace`, `runtime_ref` | TrainJobs observed after the controller's initial informer listing. |
+| `kubeflow_trainer_trainjob_complete_total` | Counter | `runtime_ref` | TrainJobs observed completing successfully. |
+| `kubeflow_trainer_trainjob_failed_total` | Counter | `reason`, `runtime_ref` | TrainJobs observed failing. Known failure reasons are preserved; unknown reasons are reported as `other`. |
+| `kubeflow_trainer_trainjob_lifetime_seconds` | Histogram | `result`, `runtime_ref` | Time from TrainJob creation to its terminal condition. |
+| `kubeflow_trainer_trainjob_requested_training_nodes_per_job` | Histogram | `runtime_ref` | Logical training nodes requested by each TrainJob after runtime defaults are applied. |
+
+`runtime_ref` is a normalized identifier in the form
+`api-group/kind/namespace/name`. The namespace component is empty for a
+`ClusterTrainingRuntime`.
+
+## Important limitations
+
+- Creation, completion, and workload metrics are event-based and reset when
+  the controller process restarts or leadership changes. Existing objects
+  found during the initial informer listing are not counted as newly created.
+- Runtime inventory is evaluated at scrape time, so it reflects the current
+  runtime resources.
+- All Trainer metric families are exposed only by the elected leader replica;
+  follower replicas expose none of them.
+- The metrics do not currently provide durable history, checkpoint usage,
+  progress, suspend/resume, accelerator utilization, or per-replicated-job
+  runtime instrumentation.
+- Labels use normalized identifiers and bounded categories, but total
+  cardinality still grows with the number of runtimes and namespaces. Avoid
+  adding job names, UIDs, raw errors, or other unbounded values.
+
+## Example PromQL
+
+Count successfully completed TrainJobs by runtime:
+
+```promql
+sum by (runtime_ref) (
+  rate(kubeflow_trainer_trainjob_complete_total[15m])
+)
+```
+
+Count failed TrainJobs by reason and runtime:
+
+```text
+sum by (reason, runtime_ref) (
+  increase(kubeflow_trainer_trainjob_failed_total[$__range])
+)
+```
+
+Calculate the observed failure percentage:
+
+```text
+100 *
+sum(increase(kubeflow_trainer_trainjob_failed_total[$__range]))
+/
+(
+  sum(increase(kubeflow_trainer_trainjob_complete_total[$__range]))
+  +
+  sum(increase(kubeflow_trainer_trainjob_failed_total[$__range]))
+)
+```
+
+Inspect the distribution of requested training nodes:
+
+```promql
+histogram_quantile(
+  0.95,
+  sum by (le, runtime_ref) (
+    rate(kubeflow_trainer_trainjob_requested_training_nodes_per_job_bucket[15m])
+  )
+)
+```
+
+The controller metrics are intended for cluster-local observability. Any
+export, retention, filtering, or aggregation into an external telemetry
+system should be reviewed and configured by the platform deployment.
+
+## Metric stability
+
+Metric names, labels, and semantics are part of the public observability
+contract. Changes will follow the [Kubernetes metrics deprecation
+policy](https://kubernetes.io/docs/reference/deprecation-policy/).

go.mod

@@ -10,6 +10,7 @@ require (
 	github.com/onsi/ginkgo/v2 v2.33.0
 	github.com/onsi/gomega v1.44.0
 	github.com/open-policy-agent/cert-controller v0.16.0
+	github.com/prometheus/client_golang v1.25.0
 	go.uber.org/zap v1.28.0
 	golang.org/x/crypto v0.58.0
 	google.golang.org/grpc v1.84.0
@@ -73,14 +74,14 @@ require (
 	github.com/grpc-ecosystem/grpc-gateway/v2 v2.29.0 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
 	github.com/json-iterator/go v1.1.12 // indirect
+	github.com/kylelemons/godebug v1.1.0 // indirect
 	github.com/mattn/go-isatty v0.0.20 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
 	github.com/pelletier/go-toml v1.9.5 // indirect
 	github.com/pkg/errors v0.9.1 // indirect
 	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
-	github.com/prometheus/client_golang v1.25.0 // indirect
 	github.com/prometheus/client_model v0.6.3 // indirect
 	github.com/prometheus/common v0.72.0 // indirect
 	github.com/prometheus/procfs v0.22.0 // indirect

pkg/config/config_test.go

@@ -1024,7 +1024,7 @@ tls:
 
 			// TLS options from the config file are applied to the webhook server.
 			// The metrics server receives its TLS config via tlsconfig.Apply inside
-			// SetupServer, called after certs are guaranteed present.
+			// Setup, called after certs are guaranteed present.
 			webhookSrv, ok := options.WebhookServer.(*webhook.DefaultServer)
 			if !ok {
 				t.Fatal("Expected WebhookServer to be a *webhook.DefaultServer")