[rickyrombo]

Summary

Keeps #253's approach of loading snapshot data into non-durable (UNLOGGED) tables, and fixes how it fails. #253 fixed out-of-memory kills during restore, and this load may be what did it. Tables that earlier restores left UNLOGGED are repaired at startup.

Why

Since #253, restores set every table in public to UNLOGGED for the data load, then back to LOGGED. Any table that failed to convert back was only logged as a warning, and the restore carried on.

• sla_node_reports stayed UNLOGGED after every restore. The LOGGED pass ran in alphabetical order. sla_node_reports references sla_rollups and sorts first, and Postgres refuses to make a table LOGGED while it references an UNLOGGED one (could not change table "sla_node_reports" to logged because it references unlogged table "sla_rollups", reproduced in the harness). This happened even with plenty of disk.
• Large tables could stay UNLOGGED for lack of disk. SET LOGGED rewrites the table and writes all of it to WAL, so it needs about twice the table's size free. An operator hit this with core_transactions (~60 GB).
• UNLOGGED tables get emptied. Crash recovery empties them, and the container entrypoint doesn't shut Postgres down cleanly, so every container stop wiped them. sla_node_reports feeds SLA rollup validation (isValidRollup), so an affected node disagrees with the network about rollups.
• It also rewrote mediorum's tables for nothing. Mediorum's tables (uploads, blobs, …) share public, so they were rewritten twice on every restore.

The first version of this PR removed the UNLOGGED load entirely. Ray confirmed #253 fixed the out-of-memory kills he was seeing during restores. #253 changed several things at once and we can't tell which one fixed them, so this version keeps the UNLOGGED load.

Changes

• Only snapshot tables go UNLOGGED for the load. Mediorum's and the ETL's tables aren't touched.
• Both conversions handle references. A table that can't convert yet because of a table it references (or that references it) is retried after the others. A table that can't go UNLOGGED at all just loads LOGGED, which is slower but safe, with a warning.
• The disk is checked before converting back. The restore requires about 2× the largest unlogged snapshot table to be free on Postgres's data disk (SHOW data_directory plus statfs), and fails with the sizes if it isn't. If free space can't be read (for example, Postgres runs on another host), it logs a warning and converts anyway.
• The restore fails if any snapshot table stays UNLOGGED, and the error shows in the restore process status.
• Startup repair: before the CometBFT node starts, UNLOGGED tables in public up to 1 GiB are converted back automatically. Larger ones are logged as errors with the command to run and the disk it needs. Nothing here fails startup.

Risks

• A failed conversion now fails the restore, which returns RETRY. CometBFT then re-runs the restore, so on a too-small disk it loops until space is freed, instead of "succeeding" with tables that get wiped on the next stop. That's visible in logs and the process status, but it repeats a long restore each time.
• The disk check sits after the data load, because that's when table sizes are known. A disk that's too small is only detected after the load.
• Startup changes persistence on existing nodes, limited to tables ≤1 GiB and run before the node starts, so nothing is writing to them. sla_node_reports is small.

Testing

• New tests (behind TEST_DB_URL, run in the harness):
  • LOGGED and UNLOGGED conversion of a parent/child pair in either order,
  • reporting tables that can't be converted,
  • snapshot-table filtering,
  • the headroom check,
  • converting back leaves non-snapshot tables alone,
  • the startup repair.
• The whole pkg/core/server package passes in the harness with TEST_DB_URL set.
• make test-unit passes.
• Not run: a full prod-sized restore.

Operators

Nodes that state-synced since #253 likely have sla_node_reports, and possibly other tables, UNLOGGED:

SELECT relname, pg_size_pretty(pg_total_relation_size(oid)) FROM pg_class WHERE relkind = 'r' AND relpersistence = 'u';

After upgrading, small tables are fixed at startup. Larger ones show up in the logs with instructions.

🤖 Generated with Claude Code