Skip to content

fix(storage): say why a repair cycle walked instead of reading the presence store - #580

Open
rickyrombo wants to merge 1 commit into
mainfrom
fix/presence-store-readiness-visibility
Open

fix(storage): say why a repair cycle walked instead of reading the presence store#580
rickyrombo wants to merge 1 commit into
mainfrom
fix/presence-store-readiness-visibility

Conversation

@rickyrombo

Copy link
Copy Markdown
Contributor

Summary

blob_presence rows survive restarts. The decision to read them does not: every cycle re-runs presenceStoreReady, and any failed gate falls back to a full walk with the reason logged once at Debug. From outside, a node that spent a day enumerating and is doing it again after a restart looks the same whether the store is disabled, a bucket was never fully enumerated, or the liveness sample rejected the disk. That is what happened on audius.rickyrombo.com on 2026-09-10, and there was no way to tell which gate failed without shell access.

Changes

  • presenceSourceForCycle now owns the per-cycle decision. It publishes a PresenceStoreStatus and returns whether the store is in use.
  • Health check gains storage.presenceStore: {enabled, used, reason, checkedAt}. reason is the gate's error text ("archive has never been walked", "17 of 64 sampled keys missing under \"/archive-blobs\"", "cleanup cycle always enumerates its buckets", …), empty when the store is in use. Nil until the first cycle of the process decides.
  • Warn instead of Debug when the store is enabled but unusable. With it off (the default) enumerating is the expected path and stays at Debug.
  • dropFromBucket forgets the row it deletes. Archive eviction and relocation delete through it, and it only cleared the in-memory cache. A row left behind for a blob that is gone is exactly the drift the liveness sample exists to catch; enough of them would send every later cycle back to a full walk.

No behavior change to which path a cycle takes.

Follow-up

The console storage page shows the walk via storagev1.PresenceWalk. Surfacing reason there too needs a proto field and regeneration, so it is left for a separate PR.

Test plan

  • make test-mediorum passes
  • TestPresenceSourceForCycleReportsWhyItWalked: disabled, cleanup, never-walked, and ready each publish the expected used/reason
  • TestDropFromBucketForgetsPresence: a bucket-scoped delete removes the store row

🤖 Generated with Claude Code

…esence store

The blob_presence rows survive restarts; the decision to read them does not.
Every cycle re-runs presenceStoreReady, and any failed gate -- store disabled,
a bucket never fully enumerated, or the liveness sample rejecting the disk --
falls back to a full walk with the reason logged once at Debug. From outside,
a node that just spent a day enumerating and is doing it again after a restart
is indistinguishable from one with the store turned off.

Move the decision into presenceSourceForCycle, which publishes a
PresenceStoreStatus (enabled, used, reason, checkedAt) on the health endpoint
and logs the fallback at Warn when the operator turned the store on. With it
off, the default, enumerating is the expected path and stays at Debug.

Also make dropFromBucket forget the row it deletes. Archive eviction and
relocation delete through it, and a row left behind for a blob that is gone is
exactly the drift the liveness sample exists to catch; enough of them would
send every later cycle back to a full walk.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant