Skip to content

Reap orphaned processes adopted by the wrapper - #351

Closed
Sayan- wants to merge 2 commits into
mainfrom
hypeship/wrapper-reap-orphans
Closed

Reap orphaned processes adopted by the wrapper#351
Sayan- wants to merge 2 commits into
mainfrom
hypeship/wrapper-reap-orphans

Conversation

@Sayan-

@Sayan- Sayan- commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Summary

The wrapper is pid 1 in both the container (ENTRYPOINT) and the unikernel (Kraftfile cmd). That makes it the machine's reaper: any process whose parent exits before it does gets reparented onto the wrapper. Nothing waited on those, so every one stayed a zombie holding a pid slot until the instance died.

Chromium relaunches are the usual source, one crashpad handler each. An instance running normally never notices. One whose services crash-loop accumulates them without bound, and the cost is not local to the browser: every /proc walk on the host scales with the pid count, so anything else sharing that kernel gets slower too. A long-lived instance in a restart loop was observed holding roughly 147k zombie pids.

Why not wait4(-1)

The obvious reaper is a SIGCHLD handler calling wait4(-1, WNOHANG) in a loop, but that is unsafe here. os/exec waits on a specific pid; if the reaper collects one of our own children first, that Cmd.Wait fails with ECHILD and its exit status is lost. main blocks on supervisord's exit status to decide when the instance is done, so losing that race would change shutdown behavior.

Instead the wrapper records the commands it starts and only ever waits on a pid it does not own. Zombie children are found by scanning /proc, and the ownership lock is held across Start so a concurrent reap cannot observe a child in the window between fork and registration.

Ownership is keyed by command identity, not by a bare pid flag. Cmd.Wait reaps the process before waitOwned can take the lock, which frees the pid for reuse, so a release that lands late would otherwise evict a newer command's entry and expose it to the reaper. releaseOwned only deletes while the pid still maps to the same command.

This adds startOwned / waitOwned / runOwned and routes the wrapper's existing exec.Command call sites through them. No behavior change at those call sites.

An init process such as tini would also solve the pid-1 problem, but only on the ENTRYPOINT path. The unikernel boots /wrapper directly through the Kraftfile, so fixing it in the binary covers both without adding a new component to the VM boot path.

Testing

go build, go vet, gofmt and go test -race -count=3 pass on server/cmd/wrapper.

reap_test.go covers the three properties that matter:

  • an unowned zombie child gets collected, which is the crashpad case
  • an owned zombie child is left alone and its Wait still returns cleanly, which is the regression guard against the wait4(-1) approach above
  • a stale release does not evict a later holder of the same pid

Each was checked against a deliberately broken implementation to confirm it fails when the property is violated, rather than passing vacuously.

Not yet exercised in a real image build. Worth a boot test on both profiles before this leaves draft.

Note for review

Stacked on #350, which fixes the one leak the wrapper causes directly rather than adopts. Both touch supervisord.go. Review that one first.


Note

Medium Risk
Changes pid-1 process lifecycle and supervisord wait paths; incorrect reaping could still break shutdown or leak zombies, though tests guard the main race cases.

Overview
Adds a pid-1 orphan reaper so adopted zombie children (e.g. Chromium crashpad handlers after relaunches) are collected instead of accumulating until instance death and slowing host /proc walks.

Reaping uses SIGCHLD plus a 30s ticker and scans /proc for zombie children, but only Wait4s pids not registered as “owned” — avoiding a global wait4(-1) that would race with os/exec and break supervisord shutdown semantics (ECHILD / lost exit status).

Introduces startOwned / waitOwned / runOwned with lock-protected registration at fork time and releaseOwned keyed by command identity so stale releases cannot drop a newer pid holder. main starts the reaper before any forks and routes supervisord lifecycle through waitOwned; supervisord.go routes log tail and runStream through the owned helpers.

reap_test.go covers unowned zombie collection, owned zombies left for Wait, and safe release when pids are reused.

Reviewed by Cursor Bugbot for commit 339f194. Bugbot is set up for automated code reviews on this repo. Configure here.

@Sayan-
Sayan- marked this pull request as ready for review August 24, 2026 20:54
tailFile started tail -F and never waited on it. The wrapper runs as pid 1
in both the container and the unikernel, so every exited tail stayed a
zombie for the life of the instance.

A scan that ends on a read error rather than EOF leaves tail alive, so kill
it before waiting instead of parking the goroutine forever.
@Sayan-
Sayan- force-pushed the hypeship/wrapper-tail-wait branch from 47c35bf to cf73a4d Compare August 24, 2026 21:00
@Sayan-
Sayan- force-pushed the hypeship/wrapper-reap-orphans branch from 6d18369 to 929d66a Compare August 24, 2026 21:00
@Sayan-
Sayan- marked this pull request as draft August 24, 2026 21:01
@Sayan-
Sayan- force-pushed the hypeship/wrapper-reap-orphans branch from 929d66a to eb1a17d Compare August 24, 2026 21:07
@Sayan-
Sayan- marked this pull request as ready for review August 24, 2026 21:10
@Sayan-
Sayan- force-pushed the hypeship/wrapper-reap-orphans branch from eb1a17d to 35b2ae7 Compare August 24, 2026 21:17
@Sayan-
Sayan- marked this pull request as draft August 24, 2026 21:24
@Sayan-
Sayan- marked this pull request as ready for review August 24, 2026 21:27
The wrapper is pid 1 in both the container and the unikernel, so any
process whose parent exits first gets reparented onto it. Nothing waited
on those, so each one stayed a zombie holding a pid slot until the
instance died. Chromium relaunches are the common source, via their
crashpad handlers.

Reaping cannot simply wait4(-1): os/exec waits on a specific pid, and
losing that race drops the exit status main relies on to know when
supervisord is done. Track the commands we start and only collect pids we
don't own, keyed by command identity so a pid freed by Wait and reused
before the release lands can't evict the new holder's entry.
@Sayan-
Sayan- force-pushed the hypeship/wrapper-reap-orphans branch from 35b2ae7 to 339f194 Compare August 24, 2026 21:31
Base automatically changed from hypeship/wrapper-tail-wait to main August 25, 2026 16:06
@Sayan-

Sayan- commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

seems overly complicated relative to problem statement. skipping for now

@Sayan- Sayan- closed this Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant