Move the blob tree off the block volume onto the system disk - #65
Merged
Conversation
The 100 GB Hetzner volume held 13 GB, and 4.2 GB of that was empty directories: ActiveStorage's DiskService creates key[0..1]/key[2..3]/ shards on write and never removes them on purge, so every snapshot deleted since 2023 left two behind. `df -i` showed 1,168,972 inodes against ~110k real files. storage:reap cannot reclaim them -- it works from database rows and only ever deletes files. What is actually on the volume is 4.3 GB of firmware assets re-downloaded hourly from GitHub, 3.6 GB of git mirrors of public repos, and 0.34 GB of live blobs on a two-day retention. The system disk has 52 GB free, so the volume buys nothing but a line on the invoice. storage/ is the only path that was pinned to the mount point. It moves to /srv/www/shared/storage, beside dev-storage, files and dl. The purge and reap containers bind-mount the same path and move with it. /srv/github-releases and /srv/github-mirror are host symlinks that become real directories, so nothing in this repo refers to them by mount point. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFwmYHCbci2esgc8AMmzJR
PR Summary by QodoMove ActiveStorage disk mounts to /srv/www/shared/storage (retire block volume)
AI Description
Diagram
High-Level Assessment
Files changed (3)
|
Code Review by Qodo
1.
|
![qodo-free-for-open-source-projects[bot]](https://avatars.githubusercontent.com/in/1068000?s=60&v=4){kind=small}
Purging a snapshot removes its blob files but not the two directories DiskService created to hold them, so the tree gains dead directories at exactly the rate it loses blobs. Since 2023 that reached 1.04 million of them -- 4.2 GB, and slow enough to walk that `du` on the tree took minutes on network storage. They were swept by hand during the volume migration. This keeps them swept: the nightly purge is the right place, because it is the thing that creates them. Two passes, because emptying a leaf makes its parent empty. The blob root moves to a variable so the mount and the sweep cannot drift apart. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFwmYHCbci2esgc8AMmzJR
The blob tree used to be a mount point, so it always existed. As a plain host directory it does not, on a rebuilt host or after a restore -- and Docker creates a missing bind-mount source itself, owned by root. The container runs as uid 1000, so every upload and every purge then fails with EACCES while the health check still reports the container up. deploy.sh creates it with the right ownership before anything mounts it, per environment, and refuses to deploy if it exists owned by someone else. purge-snapshots.sh does the same, because cron reaches the host before a deploy does. RESTORE.md gains the install -d step it was missing; the recursive chown alone cannot help a directory that is not there yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFwmYHCbci2esgc8AMmzJR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The 100 GB Hetzner volume attached to
webber-euheld 13 GB, and 4.2 GB of that was empty directories.ActiveStorage's
DiskServicecreateskey[0..1]/key[2..3]/shard directories on write and never removes them on purge, so every snapshot deleted since 2023 left two behind.df -ireported 1,168,972 inodes against ~110k real files; sampling 20 of the 1,296 top-level shards found 16,079 directories of which 16,033 were empty.storage:reapcannot reclaim these — it works from database rows and only ever deletes files.What the volume actually holds:
github-releasesgithub-mirrorstorageThe system disk has 52 GB free, so the volume buys nothing but a line on the invoice.
What changes here
storage/was the only path pinned to the mount point. It moves to/srv/www/shared/storage, besidedev-storage,filesanddl. The purge and reap containers bind-mount the same path and move with it./srv/github-releasesand/srv/github-mirrorare host symlinks that become real directories, so nothing in this repo referred to them by mount point.Host side
The data has already been copied and the empty directories swept. Once this merges,
/srv/www/deploy-srcgets agit pulland prod restarts on the new mount; the volume stays attached and untouched as a rollback until it is detached from the console.🤖 Generated with Claude Code
https://claude.ai/code/session_01MFwmYHCbci2esgc8AMmzJR