Ingot graceful shutdown: bounded drain, correct teardown ordering, no lost acked writes #30

Goal

Define and implement correct graceful shutdown for both the daemon (ingot serve) and the embedded-library mode. Shutdown roughly works today, but has concrete gaps around teardown ordering, deadlines, and error propagation, and it silently leans on crash recovery (#19) for anything caught in-flight. This issue is the requirements + the fixes.

Current state

• Server.Stop (server.go:223): api.ShutDown() then backend.Drain(ctx), aggregating errors.
• api.ShutDown() → versitygw → fiber ShutdownWithTimeout(shutDownDuration) where shutDownDuration = 10s (versitygw/s3api/server.go:43): a graceful in-flight HTTP drain, but a fixed 10s not tied to ingot's shutdown deadline.
• Backend.Drain (s3frontend/backend.go:140) → Coordinator.Close: seals the open segment, drains the flush queue (ships CARs), advances per-bucket forge_root_cid. Stated contract: "after Drain returns cleanly, no acked write is unrepresented in Postgres."
• Daemon (cmd/serve.go:69): app.Run() blocks on SIGINT/SIGTERM, then runs fx OnStop. fx.StopTimeout is not set → defaults to 15s for the entire OnStop sequence.
• Embedded mode: the host drives ServerModule's OnStop.

Requirements

R1 — Teardown ordering: quiesce listener → drain in-flight → drain log → close shared deps.
Shared dependencies (the pgx pool) must close strictly after Drain.

• Suspected bug: the pool.Close() OnStop hook (cmd/serve.go:135) is appended after the server lifecycle hook (module.go:160, via ServerModule). fx runs OnStop hooks in reverse append order, so pool.Close() runs before Server.Stop → Backend.Drain → Coordinator.Close — which needs Postgres to advance forge_root_cid and drain the flush queue. Result: on shutdown, Drain's Postgres writes hit a closed pool. Fix the ordering (make the pool a dependency the server hook tears down before, or register it so it stops last) and add a regression test that asserts the order.

R2 — Bounded in-flight drain, one knob.
In-flight requests get a bounded grace window, then a forced close. Today there are three independent, unaligned timeouts: fiber's fixed 10s, fx StopTimeout (default 15s, unset), and no ingot-level knob. Collapse to a single configurable grace period, and guarantee the log-drain sub-deadline < process StopTimeout so Drain is never cut off mid-ship.

R3 — No acked write may be lost.
Every request that returned 200 must be committed in Postgres after a clean shutdown (Drain's contract). If Drain cannot finish within the window, it must fail loudly (surface a non-zero exit), not silently drop.

R4 — Ship is safe-to-abandon; commit is not.
Un-shipped-but-locally-durable catalog CARs are re-enqueued by logstore.Open on restart, so exceeding the Forge-ship deadline should abandon-and-log, not hang and not lose data. The hard guarantee is the Postgres commit; the Forge ship (and forge_root_cid advance) may complete after restart. Make this explicit so a slow/unreachable Sprue at shutdown can't wedge the process.

R5 — In-flight write/commit + multipart invariants.
A PUT past its commit critical section must ACK; one before ACK may be abandoned, leaving spooled/parked blobs for crash recovery (#19) — acceptable, but must be documented, not incidental. Shutdown must not leave a multipart session stuck in completing/aborting (the single-winner latch invariant, §7.3).

R6 — Signals.
First SIGINT/SIGTERM → graceful drain; second signal → force-quit now (define + implement). Signal handling is daemon-only; in embedded-library mode ingot installs no signal handlers (the host owns signals) — verify this holds.

R7 — Idempotent / re-entrant teardown.
Stop/Close safe against double-invoke; no panic on double-close of the log or pool.

R8 — Error propagation + observability.
Surface a non-zero process exit when shutdown wasn't clean — verify whether app.Run() currently swallows OnStop errors (its return is ignored at cmd/serve.go:69). Log shutdown phases (listener quiesced, N in-flight drained, segment sealed, M segments shipped) with timings and a final clean vs forced outcome (today: a single "shutting down" line).

R9 — Readiness (nice-to-have).
When a health/readiness endpoint exists, flip it to not-ready at drain start so a load balancer stops routing before the listener quiesces.

Acceptance criteria

• fx OnStop order provably runs Drain before pool.Close() (regression test).
• Single configurable grace period; documented defaults; log-drain deadline < StopTimeout.
• SIGTERM during an in-flight PUT → either 200 and present in Postgres, or a clean pre-ACK rejection; never a 200 without a commit.
• SIGTERM with un-shipped catalog segments → clean exit; restart re-ships them and forge_root_cid converges.
• Non-zero exit code on an incomplete drain.
• Second signal forces immediate exit.

Refs

• Code: server.go:223 (Stop), s3frontend/backend.go:140 (Drain), module.go:160 (registerServerLifecycle), cmd/serve.go:69,135 (app.Run + pool.Close), logstore Store.Close / bucketop Coordinator.Close.
• Docs: docs/architecture.md §7.1 (write path / ACK durability), §7.5 (failure modes).
• Related: Crash recovery: implement Recover() + upload_intents × blob_refs reconciliation #19 (crash recovery cleans up whatever a non-graceful stop leaves — graceful shutdown minimizes that surface), Durability: reconcile blob_refs atomically with the catalog/root commit #18 (commit atomicity).

Part of #17