Enabling tls.enabled breaks LAPI (DaemonSet agent, default tlsClientAuth): empty agents_allowed_ou rejects the agent cert OU; plus a stale user/password conflict when migrating an existing install #355

Summary

Setting tls.enabled: true on an existing (previously non-TLS) install — with the default cert-manager integration and default agent.tlsClientAuth: true — leaves LAPI in CrashLoopBackOff and all agents stuck Init (wait-for-lapi). I hit two distinct failures, in order:

1. (migration) stale credentials conflict — user/password authentication and TLS authentication are mutually exclusive
2. (looks general) OU mismatch — LAPI rejects its own agent client cert because agents_allowed_ou is empty, while the chart issues the agent cert with OU=agent-ou. There is no values key to set the allowed OU.

Environment

• Chart: crowdsec 0.24.0 (appVersion v1.7.8, image crowdsec:v1.7.8-63227459-docker)
• Kubernetes: k3s v1.34.5
• cert-manager present; agent runs as the default DaemonSet
• LAPI uses persistent volumes (crowdsec-db, crowdsec-config) — this is an existing install being migrated to TLS

Values

Only this was added on top of an otherwise-working non-TLS install:

tls:
  enabled: true
  # certManager.enabled (default true) + empty issuerRef -> self-signed CA auto-generated
  # agent.tlsClientAuth (default true)

cert-manager objects were created correctly: Issuer/crowdsec-ca-issuer plus Certificate/{crowdsec-ca,crowdsec-lapi,crowdsec-agent,crowdsec-bouncer} (all Ready), and the corresponding *-tls secrets. The agent init container correctly switched from wait-for-lapi-and-register to wait-for-lapi (no cscli lapi register), as expected.

Failure 1 — stale creds conflict (migration only)

LAPI fatal:

level=fatal msg="loading api client: user/password authentication and TLS authentication are mutually exclusive"

The persisted local_api_credentials.yaml (on the config PVC, from the prior non-TLS install) had both auth methods after the entrypoint added the TLS paths but left the old login/password in place:

url: https://localhost:8080
login: <redacted>            # stale, pre-TLS
password: <redacted>         # stale, pre-TLS
ca_cert_path: /etc/ssl/crowdsec-lapi/ca.crt
key_path: /etc/ssl/crowdsec-agent/tls.key
cert_path: /etc/ssl/crowdsec-agent/tls.crt

Workaround: manually remove the login/password lines from the persisted file. The entrypoint should drop them (or regenerate the file) when TLS is enabled, otherwise migrating an existing install is broken.

Failure 2 — empty agents_allowed_ou rejects the agent cert (appears general)

After fixing Failure 1, LAPI still crashloops:

level=warning msg="client certificate OU [agent-ou] doesn't match expected OU []"
level=fatal msg="crowdsec init: while initializing LAPIClient: authenticate watcher (): API error: client certificate OU [agent-ou] doesn't match expected OU []"

The chart-issued agent certificate has OU=agent-ou, but the rendered LAPI config.yaml has api.server.tls.agents_allowed_ou: [] (empty). So LAPI rejects the very client certificate the chart gave the agent. The config.yaml here was freshly regenerated by the entrypoint, which suggests the empty allowlist is what the chart/entrypoint produces — i.e. this likely affects fresh installs too, not just migrations.

There is no values key to set agents_allowed_ou (and none to set the cert OU), so this can't be fixed through Helm values — only by hand-editing the LAPI config.yaml on the PVC, which is not declarative and gets clobbered on the next sync.

Expected behavior

Setting tls.enabled: true with the default cert-manager + tlsClientAuth should produce a working agent↔LAPI mutual-TLS setup out of the box:

• LAPI's agents_allowed_ou should be populated to match the OU on the chart-issued agent cert (agent-ou), or both should be configurable via values.
• On migration, stale login/password should not be left alongside TLS cert paths in local_api_credentials.yaml.

Suggested fix

• Set api.server.tls.agents_allowed_ou (and bouncers_allowed_ou) in the LAPI config to the OUs the chart actually puts on the agent/bouncer certificates, and/or expose them as values.
• When TLS is enabled, have the entrypoint regenerate local_api_credentials.yaml (or strip login/password) instead of appending cert paths to a password-based file.

Possibly related

#309 (TLS + clientAuth init env, Deployment agent), #239 (appsec tlsClientAuth), #312 (TLS secret names hardcoded), #349 (config.yaml.local path mismatch).