diff --git a/AGENTS.md b/AGENTS.md index 1e59fe0..7e5dee5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ - **Security-first design**: Security contexts enabled by default (runAsNonRoot, drop ALL capabilities, seccomp RuntimeDefault profile). - **Service accounts**: Dedicated service accounts for each component (app, engine, tables, analytics, nginx, postgres, redis) with configurable annotations for AWS IAM roles (IRSA), GCP Workload Identity, or Azure Managed Identity. - **External Secrets Operator**: Built-in support for AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, and Azure Key Vault integration. -- **PodDisruptionBudgets (PDBs)**: Configured for all stateless components to ensure minimum availability during voluntary disruptions (node drains, upgrades). +- **PodDisruptionBudgets (PDBs)**: Configured for `app`, `engine`, `nginx`, `analytics` and `tables` — note that `tables` is stateful (PVC, ReadWriteOnce), while the Postgres and Redis StatefulSets have no PDB. All default to `maxUnavailable: 1` so voluntary disruptions (node drains, upgrades) can always proceed. Each component also accepts `minAvailable` when `maxUnavailable` is unset, but avoid it where a component runs a single replica: `minAvailable: 1` then evaluates to `disruptionsAllowed: 0` and blocks every drain. `analytics` and `tables` default to one replica. - **HorizontalPodAutoscalers (HPAs)**: Optional autoscaling for app, engine, analytics, and nginx based on CPU/memory metrics. - **NetworkPolicy**: Optional network segmentation to restrict pod-to-pod communication and enforce least-privilege networking with explicit allow rules. - **LimitRange**: Optional namespace-level resource defaults and constraints to prevent resource exhaustion. diff --git a/README.md b/README.md index 3dc1650..b58f20c 100644 --- a/README.md +++ b/README.md @@ -829,26 +829,52 @@ global: topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule -# Enable PodDisruptionBudgets -pdb: - enabled: true - app: +# PodDisruptionBudgets are enabled by default with maxUnavailable: 1 +app: + podDisruptionBudget: enabled: true - minAvailable: 2 - engine: + maxUnavailable: 1 +engine: + podDisruptionBudget: enabled: true - minAvailable: 2 - nginx: + maxUnavailable: 1 +nginx: + podDisruptionBudget: enabled: true - minAvailable: 1 - analytics: + maxUnavailable: 1 +analytics: + podDisruptionBudget: enabled: true - minAvailable: 1 - tables: + maxUnavailable: 1 +tables: + podDisruptionBudget: enabled: true - minAvailable: 1 + maxUnavailable: 1 +``` + +A PodDisruptionBudget resource carries only one of the two fields. The chart renders +`maxUnavailable` whenever that value is non-nil, and `minAvailable` otherwise; +`maxUnavailable: 0` counts as set and is rendered as `0`. + +Prefer `maxUnavailable`. `minAvailable: 1` on a component running a single replica +evaluates to `disruptionsAllowed: 0`, which makes the pod impossible to evict and blocks +every node drain — cluster upgrades, node image upgrades and autoscaler scale-down all +fail while it is set. `analytics` and `tables` default to one replica, and `tables` uses +`ReadWriteOnce` storage so it cannot be scaled out of the problem. + +To use `minAvailable` instead, set `maxUnavailable` to `null` explicitly: + +```yaml +app: + podDisruptionBudget: + maxUnavailable: null + minAvailable: 2 ``` +Note that `maxUnavailable: 1` allows only one pod down at a time. At three or more +replicas that is stricter than `minAvailable: 1`, which permits all but one to go at +once — safer, but drains take longer. + ### Monitoring and observability **Prometheus metrics:** diff --git a/chart/templates/pdb-analytics.yaml b/chart/templates/pdb-analytics.yaml index 369f90d..87245ad 100644 --- a/chart/templates/pdb-analytics.yaml +++ b/chart/templates/pdb-analytics.yaml @@ -9,7 +9,11 @@ metadata: {{- toYaml . | nindent 4 }} {{- end }} spec: - minAvailable: {{ .Values.analytics.podDisruptionBudget.minAvailable }} + {{- if not (kindIs "invalid" .Values.analytics.podDisruptionBudget.maxUnavailable) }} + maxUnavailable: {{ .Values.analytics.podDisruptionBudget.maxUnavailable }} + {{- else }} + minAvailable: {{ .Values.analytics.podDisruptionBudget.minAvailable | default 1 }} + {{- end }} selector: matchLabels: {{- include "openops.componentSelectorLabels" (dict "root" . "component" "analytics") | nindent 6 }} diff --git a/chart/templates/pdb-app.yaml b/chart/templates/pdb-app.yaml index 12f9064..6df94f4 100644 --- a/chart/templates/pdb-app.yaml +++ b/chart/templates/pdb-app.yaml @@ -9,7 +9,11 @@ metadata: {{- toYaml . | nindent 4 }} {{- end }} spec: - minAvailable: {{ .Values.app.podDisruptionBudget.minAvailable }} + {{- if not (kindIs "invalid" .Values.app.podDisruptionBudget.maxUnavailable) }} + maxUnavailable: {{ .Values.app.podDisruptionBudget.maxUnavailable }} + {{- else }} + minAvailable: {{ .Values.app.podDisruptionBudget.minAvailable | default 1 }} + {{- end }} selector: matchLabels: {{- include "openops.componentSelectorLabels" (dict "root" . "component" "app") | nindent 6 }} diff --git a/chart/templates/pdb-engine.yaml b/chart/templates/pdb-engine.yaml index b0e670e..e3a2829 100644 --- a/chart/templates/pdb-engine.yaml +++ b/chart/templates/pdb-engine.yaml @@ -9,7 +9,11 @@ metadata: {{- toYaml . | nindent 4 }} {{- end }} spec: - minAvailable: {{ .Values.engine.podDisruptionBudget.minAvailable }} + {{- if not (kindIs "invalid" .Values.engine.podDisruptionBudget.maxUnavailable) }} + maxUnavailable: {{ .Values.engine.podDisruptionBudget.maxUnavailable }} + {{- else }} + minAvailable: {{ .Values.engine.podDisruptionBudget.minAvailable | default 1 }} + {{- end }} selector: matchLabels: {{- include "openops.componentSelectorLabels" (dict "root" . "component" "engine") | nindent 6 }} diff --git a/chart/templates/pdb-nginx.yaml b/chart/templates/pdb-nginx.yaml index d452c88..261a6d9 100644 --- a/chart/templates/pdb-nginx.yaml +++ b/chart/templates/pdb-nginx.yaml @@ -9,7 +9,11 @@ metadata: {{- toYaml . | nindent 4 }} {{- end }} spec: - minAvailable: {{ .Values.nginx.podDisruptionBudget.minAvailable }} + {{- if not (kindIs "invalid" .Values.nginx.podDisruptionBudget.maxUnavailable) }} + maxUnavailable: {{ .Values.nginx.podDisruptionBudget.maxUnavailable }} + {{- else }} + minAvailable: {{ .Values.nginx.podDisruptionBudget.minAvailable | default 1 }} + {{- end }} selector: matchLabels: {{- include "openops.componentSelectorLabels" (dict "root" . "component" "nginx") | nindent 6 }} diff --git a/chart/templates/pdb-tables.yaml b/chart/templates/pdb-tables.yaml index b3a7cfa..fe63e3b 100644 --- a/chart/templates/pdb-tables.yaml +++ b/chart/templates/pdb-tables.yaml @@ -9,7 +9,11 @@ metadata: {{- toYaml . | nindent 4 }} {{- end }} spec: - minAvailable: {{ .Values.tables.podDisruptionBudget.minAvailable }} + {{- if not (kindIs "invalid" .Values.tables.podDisruptionBudget.maxUnavailable) }} + maxUnavailable: {{ .Values.tables.podDisruptionBudget.maxUnavailable }} + {{- else }} + minAvailable: {{ .Values.tables.podDisruptionBudget.minAvailable | default 1 }} + {{- end }} selector: matchLabels: {{- include "openops.componentSelectorLabels" (dict "root" . "component" "tables") | nindent 6 }} diff --git a/chart/values.yaml b/chart/values.yaml index 139198f..ef6990a 100644 --- a/chart/values.yaml +++ b/chart/values.yaml @@ -193,7 +193,7 @@ app: # Pod Disruption Budget - ENABLED BY DEFAULT podDisruptionBudget: enabled: true - minAvailable: 1 + maxUnavailable: 1 # Horizontal Pod Autoscaler autoscaling: enabled: false @@ -229,7 +229,7 @@ engine: # Pod Disruption Budget - ENABLED BY DEFAULT podDisruptionBudget: enabled: true - minAvailable: 1 + maxUnavailable: 1 # Horizontal Pod Autoscaler autoscaling: enabled: false @@ -269,7 +269,7 @@ tables: # Pod Disruption Budget podDisruptionBudget: enabled: true - minAvailable: 1 + maxUnavailable: 1 # Horizontal Pod Autoscaler autoscaling: enabled: false @@ -332,7 +332,7 @@ analytics: # Pod Disruption Budget podDisruptionBudget: enabled: true - minAvailable: 1 + maxUnavailable: 1 # Horizontal Pod Autoscaler autoscaling: enabled: false @@ -545,7 +545,7 @@ nginx: # Pod Disruption Budget - ENABLED BY DEFAULT podDisruptionBudget: enabled: true - minAvailable: 1 + maxUnavailable: 1 # Horizontal Pod Autoscaler autoscaling: enabled: false diff --git a/docs/DEPLOY_TO_AWS_EKS.md b/docs/DEPLOY_TO_AWS_EKS.md index 3d17f15..254ca3c 100644 --- a/docs/DEPLOY_TO_AWS_EKS.md +++ b/docs/DEPLOY_TO_AWS_EKS.md @@ -751,23 +751,26 @@ nginx: # service.beta.kubernetes.io/aws-load-balancer-internal: "true" # Pod Disruption Budgets for HA -pdb: - enabled: true - app: +app: + podDisruptionBudget: enabled: true - minAvailable: 2 - engine: + maxUnavailable: 1 +engine: + podDisruptionBudget: enabled: true - minAvailable: 2 - nginx: + maxUnavailable: 1 +nginx: + podDisruptionBudget: enabled: true - minAvailable: 1 - analytics: + maxUnavailable: 1 +analytics: + podDisruptionBudget: enabled: true - minAvailable: 1 - tables: + maxUnavailable: 1 +tables: + podDisruptionBudget: enabled: true - minAvailable: 1 + maxUnavailable: 1 # Horizontal Pod Autoscaling hpa: