feat: External DBs as the only supported method for hypershell

[rh-amarin]

Summary

This PR removes CloudNativePG (CNPG) and per-gateway Deployment PostgreSQL as database provisioning options for gateways. Every gateway database now comes from one externally provisioned PostgreSQL server, and the control plane provisions each gateway's database and role from a single admin credential Secret mounted into the controller. This incorporates and simplifies the idea from #286 (jsell-rh, "provision gateway databases from an admin Secret"): no dual-provider fallback, no API-side database inventory, and TLS is verify-full everywhere with no downgrade path.

What is removed

• CNPG provider: the postgresql.cnpg.io Cluster/Database/DatabaseRole custom resources, the CNPG operator install and startup API-detection check, CNPG_VERSION, and the RBAC for those resource types.
• Deployment provider: the per-gateway PostgreSQL Deployment/PVC/Service the control plane used to create and own for each gateway, and HYPERSHELL_DATABASE_IMAGE/OPENSHELL_DATABASE_IMAGE.
• DATABASE_PROVIDER: the environment variable that selected between deployment, cnpg and external is gone. There is no provider selection left; there is exactly one database path.
• The ManagedDatabase API resource and Gateway.database_id: removed entirely from the REST/gRPC API, both SDKs, the CLI, the web console, and the database schema (with migrations dropping the column and table). The API server no longer knows about databases at all.
• The openshell-db-* in-cluster database namespaces and their garbage-collection exclusion.

What replaces it

• A single PostgreSQL admin credential Secret is staged out-of-band (by the platform team, or by External Secrets Operator, see the new deploy/components/gateway-database-admin-secret component) and mounted into the control plane at GATEWAY_DATABASE_ADMIN_DIR (default /etc/hypershell/gateway-database): host, port, user, password, sslrootcert required; dbname optional; sslmode optional but must be verify-full if present.
• The controller reads those files on every database operation, so an admin password rotation needs no controller restart, and refuses to start (gateway.ValidateAdminCredentialsDir) if the mount is missing or malformed. It never connects to the server at startup; reachability and privileges are checked at reconcile time with retries.
• Per gateway: role and database gw_<id>, GRANT gw_<id> TO <admin user> so a non-superuser admin with CREATEDB+CREATEROLE can own the database, REVOKE CONNECT FROM PUBLIC, GRANT CONNECT. Password reused from the existing tenant Secret or generated with crypto/rand; ALTER ROLE only as repair, never rotation.
• The tenant Secret openshell-gateway-db-credentials carries sslmode=verify-full and the CA bundle; the gateway Deployment mounts the CA at /etc/openshell-db/ca.crt. Both the admin and the gateway connections verify the server certificate and hostname; there is no TLS downgrade path.
• A failed cleanup on gateway delete returns an error to the retry queue and records a PostgreSQLDatabase IncompleteFinalization Event instead of logging and moving on.

UI

Removed the operational dashboard's database inventory widget and metric, the BFF's managed-database Prometheus queries, and the gateway detail page's database ID field.

Deploy, scripts, CI

• New scripts/gen-postgres-tls.sh shared by kind and OpenShift dev environments to mint a CA and serve the stand-in PostgreSQL over TLS, so local dev exercises the same verify-full path as production.
• Both dev drivers create the hypershell-gateway-database-admin Secret before deploying the controller; the hypershell-managed-db-* credentials namespaces and openshell-db seed are gone.
• e2e proof of database provisioning is now the tenant Secret's sslmode=verify-full and non-empty sslrootcert.

Specs and docs

specs/platform/openshell-gateway-database.spec.md is rewritten as the single source of truth for the new design (previously it covered three provider modes). platform-inventory, data-model, rbac-enforcement, naming-multitenancy and the web-console specs are updated to remove the CNPG/Deployment/ManagedDatabase surface.

Upgrade notes

• Breaking: clusters currently running the CNPG or Deployment provider have their in-cluster database gone once this rolls out. There is no automatic migration path; data must be moved to an externally provisioned server before or as part of the upgrade.
• The migrations are destructive: existing Gateway.database_id values and the managed_databases table are dropped with no backfill.
• Existing openshell-db-* namespaces lose their GC exclusion and become eligible for garbage collection.
• OpenShift dev environments need make openshift-down && make openshift-up because the credentials project and TLS Secret shape changed.
• Spoke/managed-cluster controllers each need the admin Secret mounted, since every controller instance provisions the gateways it owns.

Test plan

• go build ./... && go vet ./... && go test ./... for api-server, control-plane, cli, sdk-go
• pnpm run check for the TypeScript SDK, gateway-management-ui, operational-dashboard-ui, and web-console packages
• make ci-test (all shell unit tests, including the keycloak render test, fixed by rebasing onto main's fix(test): fix make unit-test-all failures  #299)
• All kustomize overlays render (deploy/kind, deploy/hub, deploy/keycloak, deploy/openshift, deploy/ibm)
• python3 scripts/check_forbidden_terms.py
• make kind-up && make kind-seed provisions a gateway against the TLS stand-in server
• OpenShift dev environment end-to-end
• Kind and OpenShift e2e CI jobs green

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Advanced

Run ID: bc56f806-1399-4996-9f68-d43a332f9d44

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[amber-review-bot]

Amber review

Status: Complete

View the submitted review.

[rh-amarin]

/pr-extend

[rh-amarin]

PR explainer video

Status: Complete

Watch the explainer video (4:55).

The video explains commit 9ebf4d7.

Outline

PR 300 explainer video - summary

Video: hypershell-300-explainer.mp4, 4.9 minutes, 10 slides, 6 chapters.

Outline (one line per section)

• Title: One external PostgreSQL for every gateway database.
• What changes: one database path provisioned from a mounted admin secret.
• What changes: databases disappear from the API, SDKs, CLI and console (breaking upgrade).
• Components: which parts are new, changed, or removed.
• Architecture: the database moves out of the cluster and out of the API (before/after).
• Key change: one admin secret, validated at startup and re-read per operation.
• Key change: each gateway gets a private role and database with connect isolation.
• Key change: verified TLS everywhere, with no downgrade path (code).
• Key change: a failed delete cleanup is retried and recorded, not forgotten.
• Recap: what to remember and what needs care on upgrade.

Left out

Generated SDK and OpenAPI code, the mechanical fan-out of removing one field across
CLI/SDKs/console, spec and doc rewording, test plumbing, dashboard widget internals,
and dev-script detail beyond the verify-full parity point.

Pronounce map words

CNPG, PostgreSQL, PostgreSQL's, TLS, API, APIs, SDK, SDKs, CLI, DDL, CA, RDS,
verify-full, database_id.

[amber-review-bot]

Verdict

This PR completes the migration to a single externally provisioned PostgreSQL server: the control plane provisions each gateway's role and database from a mounted admin Secret over a verify-full admin connection, and the ManagedDatabase API/CNPG surface is removed end to end with documented destructive migrations. The control-plane code is well structured and defensively coded around secret redaction and SQL identifier/literal quoting, but the gateway (tenant) database connection is shipped as sslmode=require while this PR's own security and platform specs state it SHALL always be verify-full with no downgrade - that contradiction is a security-relevant blocker, and there is material cross-PR coordination maintainers must resolve.

Summary

The change is a deliberate, clearly documented breaking migration: gateway roles/databases (gw_<id>) are provisioned by the controller from a mounted admin credential Secret, startup fails fast when the mount is absent or malformed, credentials are re-read per operation so rotation needs no restart, and failed deletion cleanup returns to the retry queue and records an operator-visible IncompleteFinalization Event. The admin connection correctly enforces verify-full with the mounted CA. The one substantive defect is that the tenant Secret is written with sslmode=require and no CA, which the accompanying specs explicitly forbid.

Strengths

• Admin-connection failures are classified (connErrorCategory) instead of surfacing the raw DSN, and CREATE ROLE/ALTER ROLE errors are returned redacted so the interpolated password never reaches an error string.
• SQL identifiers and the generated hex password are quoted (pgQuoteIdent/pgQuoteLiteral), and the DDL-interpolation limitation of lib/pq is documented in place.
• Tenant Secret writes use a read-then-diff reconcile (create-or-update), not create-and-ignore.
• The admin connection enforces verify-full and refuses to start on a weaker sslmode, and tests assert the mounted CA is kept verbatim.

Findings

[Critical] Spec Consistency / Security - tenant database connection is sslmode=require, but the specs mandate verify-full (components/control-plane/internal/gateway/database.go:295, :302, :317)

The tenant credentials Secret is written with tenantSSLMode = "require" and no sslrootcert, so the gateway->database connection is encrypted but the server certificate is not verified - it is exploitable by an active on-path attacker who can present a forged certificate and intercept the gateway's DB credentials and data. This directly contradicts specifications this same PR ships:

• specs/standards/security/security.spec.md: "Connections from the control plane and from gateway workloads to the gateway database server always use sslmode=verify-full with an operator-supplied CA bundle; there is no TLS downgrade setting."
• specs/platform/openshell-gateway-database.spec.md: "The sslmode in the tenant Secret and in uri SHALL always be verify-full. There is no configuration ... that lowers it for the gateway connection." The "Credentials Secret written" scenario requires the uri to carry sslmode=verify-full and sslrootcert=/etc/openshell-db/ca.crt and sslrootcert to equal the admin CA byte for byte; and "writes only verify-full into tenant Secrets."

The code comment explains the reason (the upstream OpenShell Helm chart consumes only the uri key and cannot mount a CA into the gateway pod), but that is a design decision the specs deny, and it leaves a security spec asserting a guarantee the implementation does not meet. Maintainers must pick one authoritative behavior: either project the CA into the gateway pod so the tenant connection reaches verify-full (make code match spec), or amend both specs to document the require downgrade and its MITM tradeoff. As written, the code and the shipped security guarantee disagree.

[Minor] Observability - raw admin-connection error is dropped, not logged (components/control-plane/internal/gateway/database.go:392, 531)

When openAdminConn fails, both ReconcileGatewayDatabase and DeleteGatewayDatabase return only the connErrorCategory label ("driver error redacted") and the underlying error is discarded without being logged anywhere. A lib/pq connection/TLS error carries the host, certificate, or hostname-mismatch detail an operator needs to diagnose a verify-full failure, and it does not contain the admin password (the password lives in the DSN userinfo, not in net.OpError/TLS/*pq.Error text). Consider logging the raw error at WARN in the trusted controller log while keeping the returned/status message redacted. This finding already has an open Amber inline thread (#300 (comment)); see that discussion rather than a new inline comment.

Test Diff Scrutiny

The modified pre-existing tests only drop the now-removed DatabaseId field and the removed managed-database-status widget from expectations; no assertion flips from accepting to rejecting a value, and the "absent phase is accepted" optional-field guarantee is preserved. The destructive migrations drop database_id/managed_databases; this is an intentional, documented breaking change (with rollback stubs), consistent with the PR's stated no-migration upgrade path rather than a silently tightened precondition.

Cross-PR coordination

• #185: This PR removes the ManagedDatabase resource and CNPG entirely and rewrites specs/platform/openshell-gateway-database.spec.md and data-model.spec.md around provisioning from a mounted admin Secret. #185 edits those same specs to define a periodic world-sync model with a ManagedDatabase reconciler, ManagedDatabase orphan-cleanup pipeline, hypershell.redhat.io/managed-database-id ownership labels, and CNPG Cluster health - a model that assumes ManagedDatabase persists as a first-class resource. The two data models are mutually exclusive. Maintainers must decide which is authoritative and reconcile these specs, and settle merge order, before either lands.
• #151: This PR removes Gateway.database_id/DatabaseId from the model, API, and SDKs. #151 adds current.DatabaseId != next.DatabaseId to its gateway drift-comparison and retains the entire ManagedDatabase/database_id interface as a drift-tracked spec field. This is a competing-interface/incompatible-assumption conflict: #151's re-provisioning gate depends on a field this PR deletes. Maintainers must decide merge order and whether database_id remains part of the Gateway spec and drift set.
• #311: Both PRs edit the single monotonic layout-version sequence and the same files (operational-dashboard-page.tsx LAYOUT_STORAGE_KEY, dashboard-layout-template.ts, specs/web-console/operational-dashboard.spec.md bump table, platform-inventory.spec.md). This PR removes the managed-database-status widget and the managed-databases metric and bumps the key to v39; #311 retains and continues to build on that widget and metric and bumps the key to v41. Whichever lands second either reuses/collides a layout version for different contents or ships a widget with no backing metric. Maintainers must decide merge order and reconcile the widget catalog and the layout-version sequence.

Previous concerns

• [Major] Spec Consistency - layout persistence key not bumped (prior review #300 (review), inline #300 (comment)): Addressed. At this head LAYOUT_STORAGE_KEY is hypershell.operational-dashboard.layout.v39 (packages/operational-dashboard-ui/src/pages/operational-dashboard-page.tsx:91), which matches specs/web-console/operational-dashboard.spec.md (persistence key v39, and the documented v39 bump row: "Retire managed-database-status"). The default-layout widget removal now bumps the key, so a stale saved layout referencing the removed widget is discarded.
• [Minor] Observability - raw admin-connection error dropped, not logged (inline #300 (comment)): Still present. At components/control-plane/internal/gateway/database.go:390-392 and :529-531 the failure path still returns only connErrorCategory(err) and never logs the raw error. Tracked in the existing inline thread; no new inline comment added.

Findings Summary (ordered by severity, highest first):

1. [Critical] Tenant DB connection ships sslmode=require with no CA while security.spec.md and openshell-gateway-database.spec.md (both updated in this PR) mandate verify-full with no downgrade - a security guarantee the code does not meet - Spec Consistency / Security (database.go:295, 302, 317)
2. [Minor] Raw admin-connection error is discarded and never logged, reducing diagnosability of verify-full failures - Observability (database.go:392, 531)

Convention Checklist (only conventions evaluated for this diff):

Convention	Result
No panic() in production code	Pass
Errors wrapped with fmt.Errorf context	Pass
No secrets in logs or error messages	Pass
Input validated (admin Secret / port / PEM / SSL mode)	Pass
SQL identifiers/literals quoted (injection)	Pass
SecurityContext on pod specs	Pass
Reconcile (update-or-create) pattern used	Pass
Status/Event recorded on error paths	Pass
Destructive migration documented with rollback	Pass
Layout persistence key bumped on widget-catalog change	Pass
TLS policy matches spec (verify-full, no downgrade)	Fail

[amber-review-bot]

[Critical] Spec Consistency / Security: the tenant Secret is written with sslmode=require and no sslrootcert, so the gateway->database connection is encrypted but the server certificate is not verified (an active on-path attacker can present a forged certificate and intercept the gateway's DB credentials and data). This contradicts specs shipped in this same PR: security.spec.md ("Connections from ... gateway workloads to the gateway database server always use sslmode=verify-full ...; there is no TLS downgrade setting") and openshell-gateway-database.spec.md ("The sslmode in the tenant Secret and in uri SHALL always be verify-full", the "Credentials Secret written" scenario requiring sslrootcert=/etc/openshell-db/ca.crt, and "writes only verify-full into tenant Secrets"). The comment explains the OpenShell Helm chart cannot mount a CA into the gateway pod, but that is a design decision the specs deny, leaving the security spec asserting a guarantee the code does not meet. Maintainers must choose one authoritative behavior: project the CA so the tenant connection reaches verify-full (match code to spec), or amend both specs to document the require downgrade and its MITM tradeoff.

[rh-amarin]

PR explainer video

Status: Complete

Watch the explainer video (4:39).

The video explains commit efd30fd. It has no live demo. The pull request has no live environment. For a demo: comment /pr-extend, wait until the environment comment shows this commit, then delete this comment.

Outline

PR 300 - One external PostgreSQL for every gateway database

Video: hypershell-300-explainer.mp4, 4.7 minutes, 10 slides, 6 chapters.

Outline (section by section)

1. Title - gateway databases move to one externally managed PostgreSQL server.
2. What changes - one external server, the three provider modes gone, nothing DB in cluster.
3. What changes - the API drops the managed-database resource; a breaking, no-migration upgrade.
4. Components - what is new, changed, removed, and unchanged across the solution.
5. Architecture - before/after: databases leave the cluster and leave the API.
6. Key change - exactly one provisioning model; the control plane runs the DDL itself.
7. Key change - validate the admin Secret at startup, re-read it every operation for rotation.
8. Key change - a per-gateway isolated role and database, no superuser admin required.
9. Key change - verify-full admin connection; deletes retry and record a warning, never orphan.
10. Recap - what to remember and what to do before upgrading.

Deliberately left out

• The mechanical fan-out of removing the API resource across SDKs, CLI, console, dashboard, and generated proto/OpenAPI (one decision, said once).
• Destructive schema migrations, dev TLS scripts, e2e proof, and spec/doc rewrites.

Note on accuracy

• At the head commit the tenant connection is encrypted only (require), not verify-full,
  because the upstream chart cannot mount a CA into the gateway pod; the video says so.

Pronounce map words

PostgreSQL, postgres, CNPG, PVC, TLS, SDK, SDKs, CLI, API, DDL, RDS, IBM, gw.