fix(storage): say why a repair cycle walked instead of reading the presence store - #580
Open
rickyrombo wants to merge 1 commit into
Open
fix(storage): say why a repair cycle walked instead of reading the presence store#580rickyrombo wants to merge 1 commit into
rickyrombo wants to merge 1 commit into
Conversation
…esence store The blob_presence rows survive restarts; the decision to read them does not. Every cycle re-runs presenceStoreReady, and any failed gate -- store disabled, a bucket never fully enumerated, or the liveness sample rejecting the disk -- falls back to a full walk with the reason logged once at Debug. From outside, a node that just spent a day enumerating and is doing it again after a restart is indistinguishable from one with the store turned off. Move the decision into presenceSourceForCycle, which publishes a PresenceStoreStatus (enabled, used, reason, checkedAt) on the health endpoint and logs the fallback at Warn when the operator turned the store on. With it off, the default, enumerating is the expected path and stays at Debug. Also make dropFromBucket forget the row it deletes. Archive eviction and relocation delete through it, and a row left behind for a blob that is gone is exactly the drift the liveness sample exists to catch; enough of them would send every later cycle back to a full walk. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
blob_presencerows survive restarts. The decision to read them does not: every cycle re-runspresenceStoreReady, and any failed gate falls back to a full walk with the reason logged once at Debug. From outside, a node that spent a day enumerating and is doing it again after a restart looks the same whether the store is disabled, a bucket was never fully enumerated, or the liveness sample rejected the disk. That is what happened onaudius.rickyrombo.comon 2026-09-10, and there was no way to tell which gate failed without shell access.Changes
presenceSourceForCyclenow owns the per-cycle decision. It publishes aPresenceStoreStatusand returns whether the store is in use.storage.presenceStore:{enabled, used, reason, checkedAt}.reasonis the gate's error text ("archive has never been walked","17 of 64 sampled keys missing under \"/archive-blobs\"","cleanup cycle always enumerates its buckets", …), empty when the store is in use. Nil until the first cycle of the process decides.dropFromBucketforgets the row it deletes. Archive eviction and relocation delete through it, and it only cleared the in-memory cache. A row left behind for a blob that is gone is exactly the drift the liveness sample exists to catch; enough of them would send every later cycle back to a full walk.No behavior change to which path a cycle takes.
Follow-up
The console storage page shows the walk via
storagev1.PresenceWalk. Surfacingreasonthere too needs a proto field and regeneration, so it is left for a separate PR.Test plan
make test-mediorumpassesTestPresenceSourceForCycleReportsWhyItWalked: disabled, cleanup, never-walked, and ready each publish the expectedused/reasonTestDropFromBucketForgetsPresence: a bucket-scoped delete removes the store row🤖 Generated with Claude Code