Reap orphaned processes adopted by the wrapper - #351
Closed
Sayan- wants to merge 2 commits into
Closed
Conversation
Sayan-
marked this pull request as ready for review
August 24, 2026 20:54
tailFile started tail -F and never waited on it. The wrapper runs as pid 1 in both the container and the unikernel, so every exited tail stayed a zombie for the life of the instance. A scan that ends on a read error rather than EOF leaves tail alive, so kill it before waiting instead of parking the goroutine forever.
Sayan-
force-pushed
the
hypeship/wrapper-tail-wait
branch
from
August 24, 2026 21:00
47c35bf to
cf73a4d
Compare
Sayan-
force-pushed
the
hypeship/wrapper-reap-orphans
branch
from
August 24, 2026 21:00
6d18369 to
929d66a
Compare
Sayan-
marked this pull request as draft
August 24, 2026 21:01
Sayan-
force-pushed
the
hypeship/wrapper-reap-orphans
branch
from
August 24, 2026 21:07
929d66a to
eb1a17d
Compare
Sayan-
marked this pull request as ready for review
August 24, 2026 21:10
Sayan-
force-pushed
the
hypeship/wrapper-reap-orphans
branch
from
August 24, 2026 21:17
eb1a17d to
35b2ae7
Compare
Sayan-
marked this pull request as draft
August 24, 2026 21:24
Sayan-
marked this pull request as ready for review
August 24, 2026 21:27
The wrapper is pid 1 in both the container and the unikernel, so any process whose parent exits first gets reparented onto it. Nothing waited on those, so each one stayed a zombie holding a pid slot until the instance died. Chromium relaunches are the common source, via their crashpad handlers. Reaping cannot simply wait4(-1): os/exec waits on a specific pid, and losing that race drops the exit status main relies on to know when supervisord is done. Track the commands we start and only collect pids we don't own, keyed by command identity so a pid freed by Wait and reused before the release lands can't evict the new holder's entry.
Sayan-
force-pushed
the
hypeship/wrapper-reap-orphans
branch
from
August 24, 2026 21:31
35b2ae7 to
339f194
Compare
Contributor
Author
|
seems overly complicated relative to problem statement. skipping for now |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The wrapper is pid 1 in both the container (
ENTRYPOINT) and the unikernel (Kraftfilecmd). That makes it the machine's reaper: any process whose parent exits before it does gets reparented onto the wrapper. Nothing waited on those, so every one stayed a zombie holding a pid slot until the instance died.Chromium relaunches are the usual source, one crashpad handler each. An instance running normally never notices. One whose services crash-loop accumulates them without bound, and the cost is not local to the browser: every
/procwalk on the host scales with the pid count, so anything else sharing that kernel gets slower too. A long-lived instance in a restart loop was observed holding roughly 147k zombie pids.Why not
wait4(-1)The obvious reaper is a SIGCHLD handler calling
wait4(-1, WNOHANG)in a loop, but that is unsafe here.os/execwaits on a specific pid; if the reaper collects one of our own children first, thatCmd.Waitfails with ECHILD and its exit status is lost.mainblocks on supervisord's exit status to decide when the instance is done, so losing that race would change shutdown behavior.Instead the wrapper records the commands it starts and only ever waits on a pid it does not own. Zombie children are found by scanning
/proc, and the ownership lock is held acrossStartso a concurrent reap cannot observe a child in the window between fork and registration.Ownership is keyed by command identity, not by a bare pid flag.
Cmd.Waitreaps the process beforewaitOwnedcan take the lock, which frees the pid for reuse, so a release that lands late would otherwise evict a newer command's entry and expose it to the reaper.releaseOwnedonly deletes while the pid still maps to the same command.This adds
startOwned/waitOwned/runOwnedand routes the wrapper's existingexec.Commandcall sites through them. No behavior change at those call sites.An init process such as tini would also solve the pid-1 problem, but only on the
ENTRYPOINTpath. The unikernel boots/wrapperdirectly through the Kraftfile, so fixing it in the binary covers both without adding a new component to the VM boot path.Testing
go build,go vet,gofmtandgo test -race -count=3pass onserver/cmd/wrapper.reap_test.gocovers the three properties that matter:Waitstill returns cleanly, which is the regression guard against thewait4(-1)approach aboveEach was checked against a deliberately broken implementation to confirm it fails when the property is violated, rather than passing vacuously.
Not yet exercised in a real image build. Worth a boot test on both profiles before this leaves draft.
Note for review
Stacked on #350, which fixes the one leak the wrapper causes directly rather than adopts. Both touch
supervisord.go. Review that one first.Note
Medium Risk
Changes pid-1 process lifecycle and supervisord wait paths; incorrect reaping could still break shutdown or leak zombies, though tests guard the main race cases.
Overview
Adds a pid-1 orphan reaper so adopted zombie children (e.g. Chromium crashpad handlers after relaunches) are collected instead of accumulating until instance death and slowing host
/procwalks.Reaping uses SIGCHLD plus a 30s ticker and scans
/procfor zombie children, but onlyWait4s pids not registered as “owned” — avoiding a globalwait4(-1)that would race withos/execand break supervisord shutdown semantics (ECHILD / lost exit status).Introduces
startOwned/waitOwned/runOwnedwith lock-protected registration at fork time andreleaseOwnedkeyed by command identity so stale releases cannot drop a newer pid holder.mainstarts the reaper before any forks and routes supervisord lifecycle throughwaitOwned;supervisord.goroutes logtailandrunStreamthrough the owned helpers.reap_test.gocovers unowned zombie collection, owned zombies left forWait, and safe release when pids are reused.Reviewed by Cursor Bugbot for commit 339f194. Bugbot is set up for automated code reviews on this repo. Configure here.