Skip to content

Fix Kubernetes control-plane restart after IP rotation - #2158

Open
mxschmitt wants to merge 1 commit into
apple:mainfrom
mxschmitt:upstream/k8s-stable-control-plane-endpoint
Open

Fix Kubernetes control-plane restart after IP rotation#2158
mxschmitt wants to merge 1 commit into
apple:mainfrom
mxschmitt:upstream/k8s-stable-control-plane-endpoint

Conversation

@mxschmitt

Copy link
Copy Markdown

Summary

Fix Kubernetes control-plane restarts when the container VM receives a new
IPv4 address.

Closes #2156.

What changes

  • Preserve /kind/kubeadm.conf after the initial kubeadm init.
  • Configure new single-node clusters with the node-local endpoint
    127.0.0.1:6443.
  • Keep the changing VM address only for node-local advertise and certificate
    state; kind already reconciles that state during its entrypoint.
  • Migrate legacy kube-proxy ConfigMaps that still contain an old VM address.
  • Add focused bootstrap configuration tests.

Why this works

controlPlaneEndpoint is the stable client endpoint in kubeadm. The
advertise address is per-node state. kube-proxy persists its kubeconfig in a
ConfigMap, so putting the rotating VM address there made a restart depend on
the previous allocation.

Before                         After
------                         -----
kube-proxy ── old VM IP       kube-proxy ── 127.0.0.1:6443
                 │                              │
       IP changes on restart          API server listens in same node
                 │                              │
       stale persistent client        stable client identity

The endpoint is deliberately loopback because the current plugin creates one
control-plane/worker node. A future multi-control-plane implementation should
use a real stable load balancer or virtual IP instead.

Restart flow

container k8s start
        │
        ├─ kind entrypoint updates per-node IP/certificate state
        ├─ wait for the API using admin.conf
        ├─ migrate legacy kube-proxy ConfigMap if needed
        └─ wait for kube-proxy, CoreDNS, and the node to become Ready

Validation

  • swift test --filter K8sBootstrapTests (3 tests passed)
  • Built the debug daemon and created a fresh cluster.
  • Stopped/started it across 192.168.64.2 -> 192.168.64.3.
  • Verified the node and system pods returned Ready.
  • Deliberately changed the kube-proxy ConfigMap back to the old address and
    verified k8s start migrated it to loopback.
  • Created an nginx Deployment and Service; a busybox pod fetched it over the
    Service after restart.

The fix does not require sticky IP allocation, host DNS configuration, or an
additional proxy process.

@mxschmitt

mxschmitt commented Aug 26, 2026

Copy link
Copy Markdown
Author

Long-term design question

This PR intentionally fixes the current single-node plugin with the smallest address-independent bootstrap change: new in-VM Kubernetes clients use 127.0.0.1:6443, while kind continues to reconcile the node's changing advertise address.

I would welcome maintainer guidance on the longer-term abstraction before multi-control-plane support is added. At that point, loopback would no longer be sufficient and the plugin would need a stable shared endpoint, for example:

  • a container/runtime-provided DNS identity,
  • a local API load-balancer with a dynamic backend set, or
  • a narrowly defined endpoint-reconciliation API.

The legacy kube-proxy migration in this PR is deliberately only a compatibility path; it is not intended to become a general address-rewrite mechanism.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Kubernetes control-plane restart fails after node IP rotation

1 participant