Skip to content

feat(wallet-toolbox): run StorageKnex on Postgres - #719

Open
shruggr wants to merge 33 commits into
bsv-blockchain:mainfrom
shruggr:feat/storageknex-postgres
Open

shruggr wants to merge 33 commits into
bsv-blockchain:mainfrom
shruggr:feat/storageknex-postgres

Conversation

@shruggr

@shruggr shruggr commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Lets StorageKnex run on Postgres through the knex pg client. Dialect fixes only; SQLite and MySQL behavior is unchanged except where noted.

Changes

  • 'Postgres' added to DBType, TableSettings.dbtype, the StorageClientBase settings check and specs/wallet/storage-adapter.yaml. determineDBType returns 'Postgres' for the pg dialect.
  • Migrations: bytea instead of blob, no binary(len) widening on Postgres, quoted activeStorage in the raw update.
  • Raw SQL uses ?? identifier bindings so camelCase columns are quoted (Postgres lowercases unquoted names). Boolean columns are compared with true/false instead of 1/0.
  • Inserts use RETURNING on Postgres to get the new id. Inside a transaction the insert runs in a savepoint, so a duplicate key caught by the find-or-insert helpers does not abort the caller's transaction.
  • getCount aliases the count and converts it with Number().
  • node-postgres returns int8 (satoshis, count(*)) as strings. StorageKnex sets an int8 parser on each connection its knex acquires. The process-wide pg.types defaults are not changed.
  • allocateChangeInput: under Postgres READ COMMITTED, the NOT EXISTS reservation check does not see an action batch reservation committed while the statement waited for the row lock. On Postgres only, the picked output is re-checked and the allocation is retried in a new transaction if it was reserved.
  • FOR UPDATE on the prepared BEEF epoch and sync state reads now applies to Postgres as well as MySQL.
  • KnexSessionManager merges boolean flags without integer comparisons. KnexSessionManager and KnexPaymentReplayStore (and the infra/wallet-infra copy) treat 23505 as a duplicate key.
  • All dialects: paged StorageKnex queries now ORDER BY the table key. Without it, consecutive LIMIT/OFFSET pages can return rows in different orders (on Postgres this happens in practice), so sync chunks could repeat or skip rows.

ChaintracksStorageKnex and adminStats (and the admin req review) still do not support Postgres.

Tests

  • RUNPOSTGRES=1 with POSTGRES_CONNECTION (JSON node-postgres config) adds a Postgres store to the existing Knex suites. CI runs the sharded toolbox jobs with a Postgres 17 service.
  • New Postgres-only tests cover int8 parsing, the savepoint on duplicate inserts, the allocateChangeInput reservation race and the remoting stores.
  • Some fixtures were adjusted so a second backend can run them (sequence advance after explicit-id inserts, per-test rows, migrated reader copy).

Local results (Postgres 17):

  • SQLite: 287 suites passed, 2 skipped; 3116 tests passed, 7 skipped.
  • SQLite + Postgres: 289 suites passed; 3122 tests passed, 1 skipped.
  • lint, typecheck, format:check, health:check pass.

Add 'Postgres' to DBType, TableSettings.dbtype and the remote settings
validator. Postgres dates use Date objects, as on MySQL.
- determineDBType returns 'Postgres' for the postgresql dialect without
  sending the MySQL VERSION() probe.
- The initial migration skips binary(len) widening on Postgres.
- Quote activeStorage in the 2025-02-22-001 raw update.
- Create sync_transfer_parts.bytes as bytea on Postgres.
- Return generated ids with RETURNING on Postgres.
- Alias count(*) in getCount and convert it with Number().
- Use ?? bindings for camelCase identifiers in raw SQL fragments.
- Bind spendable as a boolean in purgeData.
- Take the prepared-BEEF epoch and sync-state row locks on Postgres too.
- Add Setup.createPostgresKnex and Setup.postgresAfterCreate, which parse
  int8 values as numbers per connection.
KnexSessionManager no longer compares boolean columns with integers when
merging equal-timestamp updates. The merge result is unchanged. Treat
Postgres error 23505 as a duplicate key in the session and payment replay
stores.
Add pg as a dev dependency. RUNPOSTGRES and POSTGRES_CONNECTION add a
Postgres store wherever the suites add a MySQL store under RUNMYSQL;
missing test databases are created. Add a Postgres test for the Knex
remoting stores.
@socket-security

socket-security Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addednpm/​pg@​8.23.09910010091100

View full report

shruggr added 16 commits October 1, 2026 01:34
…gres

# Conflicts:
#	docs/reference/package-api-migrations.md
#	docs/reference/stack-facts.md
#	governance/package-release-notes.json
#	governance/repository-health/baselines.json
The MySQL driver writes Dates in the client time zone, so a client and server in
different zones shifted hoursOld by the offset.
…e names

MySQL rejects database names that end in a space.
Postgres does not keep row order across LIMIT/OFFSET queries without an
ORDER BY. Against a restored production wallet database, getSyncChunk for a
large user returned duplicate provenTxReqs and outputTagMaps and skipped
others. Paged find queries and the get*ForUser sync queries now order by the
table key; getCount clears the order before counting.
int8 parsing is private to StorageKnex; migrate uses migrate.latest; adminStats stays MySQL-only; Postgres inserts in a transaction always use a savepoint; the allocateChangeInput reservation re-check only runs on Postgres and leaves the other dialects' code unchanged.
@shruggr
shruggr marked this pull request as ready for review October 1, 2026 20:21
shruggr added a commit to b-open-io/1sat-sdk that referenced this pull request Oct 1, 2026
…ignatures

- wallet-node: StorageKnexPg (toolbox StorageKnex on a pg knex) replaces
  StoragePg. Existing databases need
  scripts/migrations/2026-10-01-storagepg-to-storageknex.sql first.
- wallet-server: createStorageServer runs the toolbox StorageServer
  standalone; RedisSessionManager is an AsyncSessionManager; the host no
  longer serves storage RPC; the accounts capacity gate and the hand-written
  RPC server are removed.
- cli: `1sat serve storage` replaces `1sat serve wallet`.
- wallet, cli: StorageClient sends binary request bodies.
- templates, actions: createSignature gets only hashToDirectlySign
  (@bsv/sdk 2.8.11 rejects data plus hashToDirectlySign).
- @bsv/wallet-toolbox(-client) resolve to @bopen-io 2.14.5
  (upstream 2.14.4 + bsv-blockchain/ts-stack#719).
- paymail: default avatar route; public profile no longer 500s on an
  empty avatar.
@sonarqubecloud

sonarqubecloud Bot commented Oct 2, 2026

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant