Skip to content

fix(airflow): the API issues service tokens while people keep Keycloak (#1047) - #1048

Merged
schwarztrinker merged 1 commit into
mainfrom
fix/1047-airflow-api-auth
Nov 12, 2026
Merged

schwarztrinker merged 1 commit into
mainfrom
fix/1047-airflow-api-auth

Conversation

@schwarztrinker

Copy link
Copy Markdown
Contributor

Fixes #1047.

What

The API server ran Airflow 3's default SimpleAuthManager, while every user the platform provisions is a FAB user. POST /auth/token therefore answered 401 {"detail":"Invalid credentials"} for nova-api (Secret airflow-nova-api), for admin (Secret airflow-admin), and for everyone else. Nova renders that as “The scheduler did not answer” and “Could not read runs”; the scheduler itself was healthy.

This puts the FAB manager back on the api-server and adds manifests/airflow-auth-manager.yaml, a ConfigMap carrying one small subclass:

  • The stock FabAuthManager.create_token only checks a password when FAB's AUTH_TYPE is AUTH_DB/AUTH_LDAP. This platform runs AUTH_TYPE=AUTH_OAUTH so people sign in through Keycloak, so the stock class refuses every password exchange — including the service accounts that have no browser.
  • StardeltFabAuthManager.create_token asks the FAB user store directly, with the same input validation. A Keycloak user has no local password, so people still sign in only through OIDC.
  • The auth manager and the module are mounted only on the api-server (the one container that serves the API), and step-airflow applies the ConfigMap before helm upgrade, so a fresh install cannot start without it.

Evidence (live cluster, before → after)

# before: the Secret's password is rejected
$ POST /auth/token {"username":"nova-api", ...}  -> 401 {"detail":"Invalid credentials"}
$ POST /auth/token {"username":"admin", ...}     -> 401 {"detail":"Invalid credentials"}
$ kubectl -n stardelt get deploy airflow-api-server ... AIRFLOW__API_AUTH__JWT_SECRET=<set>   # SimpleAuthManager

# after
$ POST /auth/token {"username":"nova-api", ...}  -> 201, token 315 chars
$ GET  /api/v2/dags                             -> 200 {"dags":[{"dag_id":"nyc_taxi_load",...}]}
$ POST /auth/token {"username":"admin", ...}     -> 201
$ POST /auth/token {"username":"nova-api","password":"wrong"} -> 401
$ GET  /api/v2/dags/pegelonline_hourly/dagRuns   -> 200, 3 runs

kubectl -n stardelt logs deploy/nova --since=4m | grep -c "sign-in was refused" → 0.

Checks

  • python3 -m unittest discover -s scripts/tests → Ran 273 tests, OK (skipped=1); the new test_airflow_auth_manager.py covers the wiring: the manager name, the mount and PYTHONPATH, the module compiles from the ConfigMap, the Makefile applies it before Helm, and the -vN name in manifest and values agree.
  • make lint → lint: OK (includes check-airflow-users).
  • helm template airflow apache-airflow/airflow --version 1.22.0 -f helm-values/airflow.yaml renders AIRFLOW__CORE__AUTH_MANAGER and the mount on the api-server only.
  • Deployed live with make step-airflow; release 38, api-server rolled cleanly.

Note for a later values edit

The ConfigMap name is versioned (airflow-auth-manager-v1). A content edit needs the suffix bumped — then the volume name changes and the pod rolls; an in-place ConfigMap edit would not.

The API server ran the Airflow 3 default SimpleAuthManager while every user
the platform provisions (admin, nova-api) is a FAB user, so POST /auth/token
answered 401 for all of them. Nova signs in as nova-api with the
airflow-nova-api Secret, got 401 on every call and showed "the scheduler did
not answer" and "Could not read runs" (platform#1047).

Run the FAB manager on the api-server only, through a small subclass: the
stock FabAuthManager refuses to exchange a password for a token while
AUTH_TYPE is AUTH_OAUTH (the Keycloak login people use), and the service
users have no other way in. The subclass asks the FAB user store directly.
Keycloak users have no local password, so OIDC stays the only way for people.

The module is a ConfigMap mounted by the api-server alone (it is the only
component that serves the API), on PYTHONPATH; step-airflow applies it before
the chart so a fresh install never starts without it.
Copilot AI lite review requested due to automatic review settings November 12, 2026 07:59

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(airflow): the Airflow 3 API rejects every provisioned user — Nova shows "The scheduler did not answer" and "Could not read runs"

2 participants