From 4b953cb2f230211b0385c0a7c46274715738df65 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marcelo=20Gon=C3=A7alves?= Date: Thu, 13 Aug 2026 12:26:34 +0100 Subject: [PATCH 1/2] Use maxUnavailable for PodDisruptionBudgets Every PDB in the chart set minAvailable, while analytics and tables default to a single replica. minAvailable: 1 against one replica evaluates to disruptionsAllowed: 0, so those pods cannot be evicted and every voluntary disruption fails: cluster upgrades, node image upgrades and autoscaler scale-down all stall while it is set. This is not theoretical. It left an Azure AKS estate on three-month-old node images across four clusters, with the pools reporting Failed and nothing alerting on it, because the drain could never complete. maxUnavailable: 1 is equivalent at two replicas and drainable at one, so all five components now default to it rather than only fixing the two that deadlock today. Overriding replicas down to 1 is a supported thing to do and should not reintroduce the deadlock. minAvailable still works when maxUnavailable is unset, so existing overrides are unaffected. A PDB may not set both; maxUnavailable takes precedence. One behaviour change to note: at three or more replicas maxUnavailable: 1 permits one pod down at a time where minAvailable: 1 permitted all but one. That is safer but makes drains slower. tables uses ReadWriteOnce storage and cannot be scaled past one replica, so it still incurs brief downtime while its node drains. This makes the drain possible, not seamless. Verified by rendering the chart: all five PDBs emit maxUnavailable: 1 with default values, and an override of maxUnavailable: null with minAvailable: 2 still emits minAvailable: 2. helm lint and both CI template steps pass. Note that CI never exercises this path, since values.ci.yaml disables PDBs. Part of OPS-4725 --- AGENTS.md | 2 +- README.md | 51 ++++++++++++++++++++++-------- chart/templates/pdb-analytics.yaml | 6 +++- chart/templates/pdb-app.yaml | 6 +++- chart/templates/pdb-engine.yaml | 6 +++- chart/templates/pdb-nginx.yaml | 6 +++- chart/templates/pdb-tables.yaml | 6 +++- chart/values.yaml | 10 +++--- docs/DEPLOY_TO_AWS_EKS.md | 27 +++++++++------- 9 files changed, 84 insertions(+), 36 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 1e59fe0..88a6057 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ - **Security-first design**: Security contexts enabled by default (runAsNonRoot, drop ALL capabilities, seccomp RuntimeDefault profile). - **Service accounts**: Dedicated service accounts for each component (app, engine, tables, analytics, nginx, postgres, redis) with configurable annotations for AWS IAM roles (IRSA), GCP Workload Identity, or Azure Managed Identity. - **External Secrets Operator**: Built-in support for AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, and Azure Key Vault integration. -- **PodDisruptionBudgets (PDBs)**: Configured for all stateless components to ensure minimum availability during voluntary disruptions (node drains, upgrades). +- **PodDisruptionBudgets (PDBs)**: Configured for all stateless components, defaulting to `maxUnavailable: 1` so voluntary disruptions (node drains, upgrades) can always proceed. Each component also accepts `minAvailable` when `maxUnavailable` is unset, but avoid it where a component runs a single replica: `minAvailable: 1` then evaluates to `disruptionsAllowed: 0` and blocks every drain. `analytics` and `tables` default to one replica. - **HorizontalPodAutoscalers (HPAs)**: Optional autoscaling for app, engine, analytics, and nginx based on CPU/memory metrics. - **NetworkPolicy**: Optional network segmentation to restrict pod-to-pod communication and enforce least-privilege networking with explicit allow rules. - **LimitRange**: Optional namespace-level resource defaults and constraints to prevent resource exhaustion. diff --git a/README.md b/README.md index 3dc1650..da628b9 100644 --- a/README.md +++ b/README.md @@ -829,26 +829,51 @@ global: topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule -# Enable PodDisruptionBudgets -pdb: - enabled: true - app: +# PodDisruptionBudgets are enabled by default with maxUnavailable: 1 +app: + podDisruptionBudget: enabled: true - minAvailable: 2 - engine: + maxUnavailable: 1 +engine: + podDisruptionBudget: enabled: true - minAvailable: 2 - nginx: + maxUnavailable: 1 +nginx: + podDisruptionBudget: enabled: true - minAvailable: 1 - analytics: + maxUnavailable: 1 +analytics: + podDisruptionBudget: enabled: true - minAvailable: 1 - tables: + maxUnavailable: 1 +tables: + podDisruptionBudget: enabled: true - minAvailable: 1 + maxUnavailable: 1 +``` + +Each component takes either `maxUnavailable` or `minAvailable` — a PodDisruptionBudget +may not set both, and `maxUnavailable` wins if you set both. + +Prefer `maxUnavailable`. `minAvailable: 1` on a component running a single replica +evaluates to `disruptionsAllowed: 0`, which makes the pod impossible to evict and blocks +every node drain — cluster upgrades, node image upgrades and autoscaler scale-down all +fail while it is set. `analytics` and `tables` default to one replica, and `tables` uses +`ReadWriteOnce` storage so it cannot be scaled out of the problem. + +To use `minAvailable` instead, unset `maxUnavailable` explicitly: + +```yaml +app: + podDisruptionBudget: + maxUnavailable: null + minAvailable: 2 ``` +Note that `maxUnavailable: 1` allows only one pod down at a time. At three or more +replicas that is stricter than `minAvailable: 1`, which permits all but one to go at +once — safer, but drains take longer. + ### Monitoring and observability **Prometheus metrics:** diff --git a/chart/templates/pdb-analytics.yaml b/chart/templates/pdb-analytics.yaml index 369f90d..85b74ba 100644 --- a/chart/templates/pdb-analytics.yaml +++ b/chart/templates/pdb-analytics.yaml @@ -9,7 +9,11 @@ metadata: {{- toYaml . | nindent 4 }} {{- end }} spec: - minAvailable: {{ .Values.analytics.podDisruptionBudget.minAvailable }} + {{- if .Values.analytics.podDisruptionBudget.maxUnavailable }} + maxUnavailable: {{ .Values.analytics.podDisruptionBudget.maxUnavailable }} + {{- else }} + minAvailable: {{ .Values.analytics.podDisruptionBudget.minAvailable | default 1 }} + {{- end }} selector: matchLabels: {{- include "openops.componentSelectorLabels" (dict "root" . "component" "analytics") | nindent 6 }} diff --git a/chart/templates/pdb-app.yaml b/chart/templates/pdb-app.yaml index 12f9064..5180d26 100644 --- a/chart/templates/pdb-app.yaml +++ b/chart/templates/pdb-app.yaml @@ -9,7 +9,11 @@ metadata: {{- toYaml . | nindent 4 }} {{- end }} spec: - minAvailable: {{ .Values.app.podDisruptionBudget.minAvailable }} + {{- if .Values.app.podDisruptionBudget.maxUnavailable }} + maxUnavailable: {{ .Values.app.podDisruptionBudget.maxUnavailable }} + {{- else }} + minAvailable: {{ .Values.app.podDisruptionBudget.minAvailable | default 1 }} + {{- end }} selector: matchLabels: {{- include "openops.componentSelectorLabels" (dict "root" . "component" "app") | nindent 6 }} diff --git a/chart/templates/pdb-engine.yaml b/chart/templates/pdb-engine.yaml index b0e670e..51b82aa 100644 --- a/chart/templates/pdb-engine.yaml +++ b/chart/templates/pdb-engine.yaml @@ -9,7 +9,11 @@ metadata: {{- toYaml . | nindent 4 }} {{- end }} spec: - minAvailable: {{ .Values.engine.podDisruptionBudget.minAvailable }} + {{- if .Values.engine.podDisruptionBudget.maxUnavailable }} + maxUnavailable: {{ .Values.engine.podDisruptionBudget.maxUnavailable }} + {{- else }} + minAvailable: {{ .Values.engine.podDisruptionBudget.minAvailable | default 1 }} + {{- end }} selector: matchLabels: {{- include "openops.componentSelectorLabels" (dict "root" . "component" "engine") | nindent 6 }} diff --git a/chart/templates/pdb-nginx.yaml b/chart/templates/pdb-nginx.yaml index d452c88..85bf478 100644 --- a/chart/templates/pdb-nginx.yaml +++ b/chart/templates/pdb-nginx.yaml @@ -9,7 +9,11 @@ metadata: {{- toYaml . | nindent 4 }} {{- end }} spec: - minAvailable: {{ .Values.nginx.podDisruptionBudget.minAvailable }} + {{- if .Values.nginx.podDisruptionBudget.maxUnavailable }} + maxUnavailable: {{ .Values.nginx.podDisruptionBudget.maxUnavailable }} + {{- else }} + minAvailable: {{ .Values.nginx.podDisruptionBudget.minAvailable | default 1 }} + {{- end }} selector: matchLabels: {{- include "openops.componentSelectorLabels" (dict "root" . "component" "nginx") | nindent 6 }} diff --git a/chart/templates/pdb-tables.yaml b/chart/templates/pdb-tables.yaml index b3a7cfa..368bf27 100644 --- a/chart/templates/pdb-tables.yaml +++ b/chart/templates/pdb-tables.yaml @@ -9,7 +9,11 @@ metadata: {{- toYaml . | nindent 4 }} {{- end }} spec: - minAvailable: {{ .Values.tables.podDisruptionBudget.minAvailable }} + {{- if .Values.tables.podDisruptionBudget.maxUnavailable }} + maxUnavailable: {{ .Values.tables.podDisruptionBudget.maxUnavailable }} + {{- else }} + minAvailable: {{ .Values.tables.podDisruptionBudget.minAvailable | default 1 }} + {{- end }} selector: matchLabels: {{- include "openops.componentSelectorLabels" (dict "root" . "component" "tables") | nindent 6 }} diff --git a/chart/values.yaml b/chart/values.yaml index 139198f..ef6990a 100644 --- a/chart/values.yaml +++ b/chart/values.yaml @@ -193,7 +193,7 @@ app: # Pod Disruption Budget - ENABLED BY DEFAULT podDisruptionBudget: enabled: true - minAvailable: 1 + maxUnavailable: 1 # Horizontal Pod Autoscaler autoscaling: enabled: false @@ -229,7 +229,7 @@ engine: # Pod Disruption Budget - ENABLED BY DEFAULT podDisruptionBudget: enabled: true - minAvailable: 1 + maxUnavailable: 1 # Horizontal Pod Autoscaler autoscaling: enabled: false @@ -269,7 +269,7 @@ tables: # Pod Disruption Budget podDisruptionBudget: enabled: true - minAvailable: 1 + maxUnavailable: 1 # Horizontal Pod Autoscaler autoscaling: enabled: false @@ -332,7 +332,7 @@ analytics: # Pod Disruption Budget podDisruptionBudget: enabled: true - minAvailable: 1 + maxUnavailable: 1 # Horizontal Pod Autoscaler autoscaling: enabled: false @@ -545,7 +545,7 @@ nginx: # Pod Disruption Budget - ENABLED BY DEFAULT podDisruptionBudget: enabled: true - minAvailable: 1 + maxUnavailable: 1 # Horizontal Pod Autoscaler autoscaling: enabled: false diff --git a/docs/DEPLOY_TO_AWS_EKS.md b/docs/DEPLOY_TO_AWS_EKS.md index 3d17f15..254ca3c 100644 --- a/docs/DEPLOY_TO_AWS_EKS.md +++ b/docs/DEPLOY_TO_AWS_EKS.md @@ -751,23 +751,26 @@ nginx: # service.beta.kubernetes.io/aws-load-balancer-internal: "true" # Pod Disruption Budgets for HA -pdb: - enabled: true - app: +app: + podDisruptionBudget: enabled: true - minAvailable: 2 - engine: + maxUnavailable: 1 +engine: + podDisruptionBudget: enabled: true - minAvailable: 2 - nginx: + maxUnavailable: 1 +nginx: + podDisruptionBudget: enabled: true - minAvailable: 1 - analytics: + maxUnavailable: 1 +analytics: + podDisruptionBudget: enabled: true - minAvailable: 1 - tables: + maxUnavailable: 1 +tables: + podDisruptionBudget: enabled: true - minAvailable: 1 + maxUnavailable: 1 # Horizontal Pod Autoscaling hpa: From 5bcde25174365e73ced453a269fa26039a66ac48 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marcelo=20Gon=C3=A7alves?= Date: Thu, 13 Aug 2026 12:36:30 +0100 Subject: [PATCH 2/2] Respect an explicit maxUnavailable of zero MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The template guarded on truthiness, and Go templates treat 0 as false, so maxUnavailable: 0 fell through to rendering minAvailable: 1 — silently producing a PDB that differs from the values that asked for it. 0 is a valid PodDisruptionBudget value. kindIs "invalid" tests for nil instead, which keeps all four cases correct: an explicit 0 renders as 0, an explicit null falls back to minAvailable, an absent key falls back, and a set value renders. hasKey would not work here, since it is true for maxUnavailable: null and would render an empty field. Also correct two documentation errors. The README claimed a PDB "may not set both" fields and then that maxUnavailable "wins if you set both", conflating the rendered resource with the values schema. AGENTS.md described PDBs as covering "all stateless components" while tables, which has one, is stateful. Part of OPS-4725 --- AGENTS.md | 2 +- README.md | 7 ++++--- chart/templates/pdb-analytics.yaml | 2 +- chart/templates/pdb-app.yaml | 2 +- chart/templates/pdb-engine.yaml | 2 +- chart/templates/pdb-nginx.yaml | 2 +- chart/templates/pdb-tables.yaml | 2 +- 7 files changed, 10 insertions(+), 9 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 88a6057..7e5dee5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ - **Security-first design**: Security contexts enabled by default (runAsNonRoot, drop ALL capabilities, seccomp RuntimeDefault profile). - **Service accounts**: Dedicated service accounts for each component (app, engine, tables, analytics, nginx, postgres, redis) with configurable annotations for AWS IAM roles (IRSA), GCP Workload Identity, or Azure Managed Identity. - **External Secrets Operator**: Built-in support for AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, and Azure Key Vault integration. -- **PodDisruptionBudgets (PDBs)**: Configured for all stateless components, defaulting to `maxUnavailable: 1` so voluntary disruptions (node drains, upgrades) can always proceed. Each component also accepts `minAvailable` when `maxUnavailable` is unset, but avoid it where a component runs a single replica: `minAvailable: 1` then evaluates to `disruptionsAllowed: 0` and blocks every drain. `analytics` and `tables` default to one replica. +- **PodDisruptionBudgets (PDBs)**: Configured for `app`, `engine`, `nginx`, `analytics` and `tables` — note that `tables` is stateful (PVC, ReadWriteOnce), while the Postgres and Redis StatefulSets have no PDB. All default to `maxUnavailable: 1` so voluntary disruptions (node drains, upgrades) can always proceed. Each component also accepts `minAvailable` when `maxUnavailable` is unset, but avoid it where a component runs a single replica: `minAvailable: 1` then evaluates to `disruptionsAllowed: 0` and blocks every drain. `analytics` and `tables` default to one replica. - **HorizontalPodAutoscalers (HPAs)**: Optional autoscaling for app, engine, analytics, and nginx based on CPU/memory metrics. - **NetworkPolicy**: Optional network segmentation to restrict pod-to-pod communication and enforce least-privilege networking with explicit allow rules. - **LimitRange**: Optional namespace-level resource defaults and constraints to prevent resource exhaustion. diff --git a/README.md b/README.md index da628b9..b58f20c 100644 --- a/README.md +++ b/README.md @@ -852,8 +852,9 @@ tables: maxUnavailable: 1 ``` -Each component takes either `maxUnavailable` or `minAvailable` — a PodDisruptionBudget -may not set both, and `maxUnavailable` wins if you set both. +A PodDisruptionBudget resource carries only one of the two fields. The chart renders +`maxUnavailable` whenever that value is non-nil, and `minAvailable` otherwise; +`maxUnavailable: 0` counts as set and is rendered as `0`. Prefer `maxUnavailable`. `minAvailable: 1` on a component running a single replica evaluates to `disruptionsAllowed: 0`, which makes the pod impossible to evict and blocks @@ -861,7 +862,7 @@ every node drain — cluster upgrades, node image upgrades and autoscaler scale- fail while it is set. `analytics` and `tables` default to one replica, and `tables` uses `ReadWriteOnce` storage so it cannot be scaled out of the problem. -To use `minAvailable` instead, unset `maxUnavailable` explicitly: +To use `minAvailable` instead, set `maxUnavailable` to `null` explicitly: ```yaml app: diff --git a/chart/templates/pdb-analytics.yaml b/chart/templates/pdb-analytics.yaml index 85b74ba..87245ad 100644 --- a/chart/templates/pdb-analytics.yaml +++ b/chart/templates/pdb-analytics.yaml @@ -9,7 +9,7 @@ metadata: {{- toYaml . | nindent 4 }} {{- end }} spec: - {{- if .Values.analytics.podDisruptionBudget.maxUnavailable }} + {{- if not (kindIs "invalid" .Values.analytics.podDisruptionBudget.maxUnavailable) }} maxUnavailable: {{ .Values.analytics.podDisruptionBudget.maxUnavailable }} {{- else }} minAvailable: {{ .Values.analytics.podDisruptionBudget.minAvailable | default 1 }} diff --git a/chart/templates/pdb-app.yaml b/chart/templates/pdb-app.yaml index 5180d26..6df94f4 100644 --- a/chart/templates/pdb-app.yaml +++ b/chart/templates/pdb-app.yaml @@ -9,7 +9,7 @@ metadata: {{- toYaml . | nindent 4 }} {{- end }} spec: - {{- if .Values.app.podDisruptionBudget.maxUnavailable }} + {{- if not (kindIs "invalid" .Values.app.podDisruptionBudget.maxUnavailable) }} maxUnavailable: {{ .Values.app.podDisruptionBudget.maxUnavailable }} {{- else }} minAvailable: {{ .Values.app.podDisruptionBudget.minAvailable | default 1 }} diff --git a/chart/templates/pdb-engine.yaml b/chart/templates/pdb-engine.yaml index 51b82aa..e3a2829 100644 --- a/chart/templates/pdb-engine.yaml +++ b/chart/templates/pdb-engine.yaml @@ -9,7 +9,7 @@ metadata: {{- toYaml . | nindent 4 }} {{- end }} spec: - {{- if .Values.engine.podDisruptionBudget.maxUnavailable }} + {{- if not (kindIs "invalid" .Values.engine.podDisruptionBudget.maxUnavailable) }} maxUnavailable: {{ .Values.engine.podDisruptionBudget.maxUnavailable }} {{- else }} minAvailable: {{ .Values.engine.podDisruptionBudget.minAvailable | default 1 }} diff --git a/chart/templates/pdb-nginx.yaml b/chart/templates/pdb-nginx.yaml index 85bf478..261a6d9 100644 --- a/chart/templates/pdb-nginx.yaml +++ b/chart/templates/pdb-nginx.yaml @@ -9,7 +9,7 @@ metadata: {{- toYaml . | nindent 4 }} {{- end }} spec: - {{- if .Values.nginx.podDisruptionBudget.maxUnavailable }} + {{- if not (kindIs "invalid" .Values.nginx.podDisruptionBudget.maxUnavailable) }} maxUnavailable: {{ .Values.nginx.podDisruptionBudget.maxUnavailable }} {{- else }} minAvailable: {{ .Values.nginx.podDisruptionBudget.minAvailable | default 1 }} diff --git a/chart/templates/pdb-tables.yaml b/chart/templates/pdb-tables.yaml index 368bf27..fe63e3b 100644 --- a/chart/templates/pdb-tables.yaml +++ b/chart/templates/pdb-tables.yaml @@ -9,7 +9,7 @@ metadata: {{- toYaml . | nindent 4 }} {{- end }} spec: - {{- if .Values.tables.podDisruptionBudget.maxUnavailable }} + {{- if not (kindIs "invalid" .Values.tables.podDisruptionBudget.maxUnavailable) }} maxUnavailable: {{ .Values.tables.podDisruptionBudget.maxUnavailable }} {{- else }} minAvailable: {{ .Values.tables.podDisruptionBudget.minAvailable | default 1 }}