Repository navigation
fix(index): give the write-ahead log's disk space back (#98) - #100
Merged
Merged
Conversation
SQLite never shrinks a -wal file by itself. After one large transaction (a schema migration, a big build) the file kept that size through every later checkpoint and restart: 1.08 GB beside a 3.6 GB index in #98. The index connection now sets journal_size_limit to 64 MB, so the first write after a checkpoint cuts the file back, and runs a TRUNCATE checkpoint at the end of open() and before close(). The checkpoint runs with busy_timeout at 0, so a sibling process mid-read or mid-build makes it return busy at once instead of stalling the open for ten seconds. Fixes #98
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #98.
SQLite never shrinks a
-walfile by itself. A checkpoint copies the frames into the database and the next writer starts again at the front of the same file, so after one large transaction (a schema migration, a big build) the log keeps that size for good. #98 measured 1.08 GB beside a 3.6 GB index, unchanged for three weeks through daily updates and restarts.Change (
src/features/search/sqlite-index.ts):PRAGMA journal_size_limit = 64 MBon the index connection (it is per connection, so every open sets it). The first write after a checkpoint cuts the file back to 64 MB.wal_checkpoint(TRUNCATE)at the end ofopen()and just beforeclose(). Closing the last connection already deletes the log, but a sibling process sharing the data directory (a host probe, a second client, anindex updaterun) makes this connection not the last one.busy_timeout = 0, restored afterwards. Under the 10 s timeout it would stall an open behind a sibling's read or build. At 0 it returnsbusy: 1at once, having copied what it could, and the next open or close reclaims the file. A failure never stops the index opening; corruption still reaches the existing sideline path.Tests (
tests/features/search-wal.test.ts, realnode:sqlitefiles, a second connection kept open so no close is the last one):journal_size_limit= 64 MB.All three fail on
main. Removing thebusy_timeout = 0line makes the second one take 10.2 s and fail.Gates:
typecheck,typecheck:tests, eslint on the touched files, and the full suite: 2731 passed. The only failures in the full parallel run were 5 s timeouts intests/cli/entry.test.tsspawningtsx, which pass alone in 1 to 2 s per case and do not touch the index.