Skip to content

fix(index): give the write-ahead log's disk space back (#98) - #100

Merged
oscardvs merged 1 commit into
mainfrom
fix/98-wal-truncate
Sep 28, 2026
Merged

oscardvs merged 1 commit into
mainfrom
fix/98-wal-truncate

Conversation

@oscardvs

Copy link
Copy Markdown
Owner

Fixes #98.

SQLite never shrinks a -wal file by itself. A checkpoint copies the frames into the database and the next writer starts again at the front of the same file, so after one large transaction (a schema migration, a big build) the log keeps that size for good. #98 measured 1.08 GB beside a 3.6 GB index, unchanged for three weeks through daily updates and restarts.

Change (src/features/search/sqlite-index.ts):

  • PRAGMA journal_size_limit = 64 MB on the index connection (it is per connection, so every open sets it). The first write after a checkpoint cuts the file back to 64 MB.
  • A wal_checkpoint(TRUNCATE) at the end of open() and just before close(). Closing the last connection already deletes the log, but a sibling process sharing the data directory (a host probe, a second client, an index update run) makes this connection not the last one.
  • The checkpoint runs with busy_timeout = 0, restored afterwards. Under the 10 s timeout it would stall an open behind a sibling's read or build. At 0 it returns busy: 1 at once, having copied what it could, and the next open or close reclaims the file. A failure never stops the index opening; corruption still reaches the existing sideline path.

Tests (tests/features/search-wal.test.ts, real node:sqlite files, a second connection kept open so no close is the last one):

  • a log grown to 8 MB by a sibling is truncated to 0 when the index opens, and the data is intact;
  • with the sibling mid-read, the open returns in well under the busy timeout and leaves the log, then the close truncates it once the read ends;
  • the index connection reports journal_size_limit = 64 MB.

All three fail on main. Removing the busy_timeout = 0 line makes the second one take 10.2 s and fail.

Gates: typecheck, typecheck:tests, eslint on the touched files, and the full suite: 2731 passed. The only failures in the full parallel run were 5 s timeouts in tests/cli/entry.test.ts spawning tsx, which pass alone in 1 to 2 s per case and do not touch the index.

SQLite never shrinks a -wal file by itself. After one large transaction
(a schema migration, a big build) the file kept that size through every
later checkpoint and restart: 1.08 GB beside a 3.6 GB index in #98.

The index connection now sets journal_size_limit to 64 MB, so the first
write after a checkpoint cuts the file back, and runs a TRUNCATE
checkpoint at the end of open() and before close(). The checkpoint runs
with busy_timeout at 0, so a sibling process mid-read or mid-build makes
it return busy at once instead of stalling the open for ten seconds.

Fixes #98
@oscardvs
oscardvs merged commit 178155d into main Sep 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Index WAL file stays at ~1.1 GB forever — consider wal_checkpoint(TRUNCATE) on open or a graceful shutdown close

1 participant