Skip to content

fix: make notebooks ready when their server listens - #1552

Open
matbun wants to merge 2 commits into
kubeflow:notebooks-v1from
matbun:fix/ready-when-server-listens
Open

matbun wants to merge 2 commits into
kubeflow:notebooks-v1from
matbun:fix/ready-when-server-listens

Conversation

@matbun

@matbun matbun commented Oct 24, 2026

Copy link
Copy Markdown

closes: #1551

Two commits. The first shows the steps of a start as waiting: only a container that will not start by itself (an error waiting reason, or terminated), an unschedulable pod, or a Warning event since the latest condition change make a warning. Readiness probe failures are part of a start and complete the waiting message. The list route lists the namespace's Notebook events once, only when needed, and an events error no longer fails it.

The second adds a readinessProbe per server type to the spawner config, as mainContainer.readinessProbe on a WorkspaceKind in Notebooks v2, applied to new notebooks; the default is tcpSocket on 8888. Existing notebooks get no probe, and a custom image not listening on 8888 stays waiting, with the probe failure as the reason.

Tested with unit tests and on our cluster (notebook controller 1.10.0, Istio 1.28): notebooks turn ready when the server answers, also with the sidecar, and one stopped while starting shows stopping, then stopped.

matbun and others added 2 commits October 24, 2026 10:27
The status showed normal steps of a start as warnings: the
PodInitializing check compared a list to a string and never matched, and
any pod condition with a reason, ContainersNotReady included, was a
warning.

Now only a container that will not start by itself (an error waiting
reason, or terminated) and an unschedulable pod are warnings; a running
container that is not ready waits for its server. A waiting notebook
turns warning when a Warning event seen since its latest condition
change explains why it is stuck, so a resolved FailedScheduling no
longer counts. Readiness probe failures are part of a start: they do
not make a warning, and the latest one completes the waiting message.

The notebooks list shares one list of the namespace's Notebook events,
listed only when a notebook needs it, and a user who may not list
events gets the status without them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matteo Bunino <48362942+matbun@users.noreply.github.com>
A notebook pod (Jupyter, VS Code or RStudio) has no readiness probe, so
the web app offers Connect as soon as the container starts, before the
server listens, and an early click gets "no healthy upstream". The
spawner config can now give each server type a readinessProbe (image,
imageGroupOne, imageGroupTwo), as mainContainer.readinessProbe does for
a WorkspaceKind in Notebooks v2, and the backend copies it onto the
container of a new notebook. The default config, in both copies, sets
tcpSocket on 8888, where the example images listen.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Matteo Bunino <48362942+matbun@users.noreply.github.com>
@github-project-automation github-project-automation Bot moved this to Needs Triage in Kubeflow Notebooks Oct 24, 2026
@google-oss-prow google-oss-prow Bot added the area/backend area - related to backend components label Oct 24, 2026
@google-oss-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign thesuperzapper for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow Bot added area/frontend area - related to frontend components area/v1 area - version - kubeflow notebooks v1 size/XL labels Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/backend area - related to backend components area/frontend area - related to frontend components area/v1 area - version - kubeflow notebooks v1 size/XL

Projects

Status: Needs Triage

Development

Successfully merging this pull request may close these issues.

1 participant