Repository navigation
[1.36.1] Count booting nodes as upcoming after CA restart to prevent duplicate scale-ups - #94
Merged
Maximiliano Uribe Falcon (MaximilianoUribe) merged 3 commits intoOct 23, 2026
Conversation
(cherry picked from commit e2dc4aad7148eb79e56901724ec9b45a0c2a2719 in kubernetes-sigs/cluster-autoscaler, adapted to the pre-context ClusterStateRegistry API and cluster-autoscaler/ module layout)
…ve scale-up request GetUpcomingNodes() previously skipped node groups without active scale-up requests or in backoff. However, because scaleUpRequests is in-memory only, real booting nodes (NotStarted) and healthy instances (Unregistered) were skipped after CA restart/failover, triggering duplicate scale-ups and premature scale-down. Partition upcoming nodes into backed instances (NotStarted + Unregistered) and the unbacked TargetSize delta. Always count backed instances, while continuing to gate the unbacked delta on having an active scale-up request and not being backed off. (cherry picked from commit 3b3e98c7ff386dc3941ea089068a44556fe58a0f in kubernetes-sigs/cluster-autoscaler, adapted to klog and the pre-context ClusterStateRegistry API)
Verify that after a CA restart a deallocate-policy node group still counts NotStarted nodes as upcoming, while deallocated nodes being started stay part of the unbacked delta gated on an active scale-up request.
Maximiliano Uribe Falcon (MaximilianoUribe)
marked this pull request as ready for review
October 23, 2026 05:59
Alex Leites (tallaxes)
approved these changes
Oct 23, 2026
Maximiliano Uribe Falcon (MaximilianoUribe)
merged commit Oct 23, 2026
a95ad12
into
cluster-autoscaler-release-1.36.1-aks
9 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this? (bug, cleanup, documentation, feature)
bug
What this PR does / why we need it?:
The 1.36.1 release includes kubernetes#9360, which stops
GetUpcomingNodes()from counting upcoming nodes for node groups that have no active scale-up request or are backed off. Scale-up requests and backoff are held only in memory, so after a cluster-autoscaler restart or leader failover, nodes that are already booting (NotStarted) or created but not yet registered (Unregistered) are no longer counted as upcoming. The pending pods that triggered the original scale-up then look unschedulable, and CA scales the node group up again.NotStartednodes can also be treated as scale-down candidates.This backports the upstream fix, kubernetes-sigs/cluster-autoscaler#182. It splits upcoming nodes into:
NotStarted+Unregistered. These always count as upcoming.We observed the duplicate scale-up with the 1.36.1 build: the CA pod restarted while a scale-up node was still provisioning, the new pod logged
1 unregistered nodes present, and it then scaled the same node group from 3 to 4. The 1.33.6, 1.34.5 and 1.35.2 branches don't include kubernetes#9360 and are unaffected.Commits:
TestGetUpcomingNodesBackedVsUnbacked,TestGetUpcomingNodesAfterRestart, and a CA-restart case inTestStaticAutoscalerUpcomingScaleDownCandidates).clusterstate.go.NotStartednodes count as upcoming. Deallocated nodes being started stay in the unbacked delta, which is unchanged behavior.Commits 1 and 2 are adapted from the upstream commits to this branch's
kloglogging, pre-contextClusterStateRegistryAPI andcluster-autoscaler/module layout. The logic is unchanged.Does this PR introduce a user-facing change?
WorkItem
N/A
Testing
go test ./clusterstate/... ./core/... ./processors/customresources/...pass, andgo test -race ./clusterstate/... ./core/ ./core/scaleup/...pass. The new regression tests fail on the current branch tip without the fix: all CA-restart cases ofTestGetUpcomingNodesBackedVsUnbacked,TestGetUpcomingNodesAfterRestart, the CA-restart case ofTestStaticAutoscalerUpcomingScaleDownCandidates, and the restart case of the new deallocate test. They pass with the fix.Cherry-pick details
Is this a cherry-picked PR?
Is this PR getting merged on upstream ?
Special notes for your reviewer:
Only cluster-autoscaler-release-1.36.1-aks needs this. The older AKS release branches don't contain kubernetes#9360. Original commits are attributed to the upstream author.
Additional Documentation: kubernetes#9360, kubernetes#9813