Improve wallet sync throughput, binary transport, and checkpoint recovery

[BraydenLangley]

Results at a glance

Smaller transfers, resumable large records, durable checkpoints, and safer proof recovery.

Scenario	Verified result
Earlier live native E2E with a large wallet	Two full copies with identical entity counts; 12m 50s vs 16m 30s (22.3% less time) between successive candidates.
Retained device backup	Completed after cancellation, connection interruption and restart; all 12 store counts preserved; repeat sync 0 inserts / 0 updates.
Large records	Synthetic 7 MiB HTTP → browser IndexedDB restore with matching hash; interruption, restart, lost acknowledgement and corruption covered.
Slow transport	Authenticated oversized-record regression also passes with 1 second added per transport send. This models latency, not bandwidth.
Proof validation	Read-only 250-proof component sample: 42.6s → 5.58s, with the same root/header checks. Stale-proof provider failover reproduced and validated using source data.

The latest revision is deployed to a compatible backup provider; its complete source-to-backup retest remains pending the primary-provider upgrade. Earlier native results do not prove this follow-up completes the reported wallet backup. Timings compare successive candidates, not a controlled comparison against upstream main; byte checks were sampled, not database-wide.

Implementation, verification, and release boundaries

Program and scope

Rebased onto upstream main at 7a9cca774a, then integrated the newly merged upstream security changes at 99f5f5faa1. This repository uses main, not master; the PR title is unchanged. Reviewed head: 4e81e20. The final follow-up changes only the measured Linux artifact allowance and documentation; runtime source is identical to the locally tested combined candidate.

• Adaptive source queries, indexed local matching, atomic page/checkpoint commits and optional compact checkpoints avoid repeated scans and large state downloads.
• Negotiated binary payloads and authenticated bounded transfers handle large records with durable staging, full-frame integrity verification, checkpoint-bound commits and replay protection. Legacy peers keep their ordinary-page formats.
• Each copy starts with 64 records and adapts after committed pages toward a five-second work budget. Proof pages cap growth at 128; metadata pages at 1,000; provider byte/item ceilings still apply.
• The server validates at most eight proofs concurrently and drains started checks on failure. Validation still checks transaction identity, Merkle membership/index/root and the active block header before persistence.
• Stale proofs may be replaced only with independently validated metadata for the same transaction. Provider traversal is bounded and isolated per lookup; custom services can omit the additive getValidatedMerklePath method. Raw transaction bytes and wallet references are preserved. Local correction timestamps enable onward incremental replication without advancing the source cursor past uncopied records.
• Unverifiable proofs reject the entire page with recovery guidance. Provider outages can still interrupt a copy; completed pages remain durable. Uncertain ordinary writes are not blindly replayed.

Impact and verification

• Public package source/types and security-sensitive authentication, serialization and persistence boundaries changed.
• Documentation, changelog, release notes and migration guidance updated.
• Self-reviewed correctness, security, compatibility, API, package artifacts, resource bounds and operational impact.
• Exact-head hosted checks passed: 38 successful checks, including merge-gate, Codecov patch coverage, CodeQL, zero-new-Sonar, conformance, mutation and platform checks; four expected scope skips accepted by the merge gate. Review threads are resolved; no open PR CodeQL alerts.

Local verification (combined sync and upstream security candidate):

• pnpm health:check, pnpm lint, pnpm format:check, pnpm typecheck.
• Toolbox: 237 suites / 2,263 passed, one existing skip. Tests include authenticated proof-page admission, independent stale-provider failover, invalid replacement rejection, drained concurrent failures, new/existing proof propagation, unchanged source cursors and checkpoint preservation.
• Focused SDK regression tests: 109 passed, including the upstream signature checks and sync authentication/hash boundaries. Client/mobile tests: 19 / 42 passed.
• All three distributions: build and packed clean-consumer checks, including conditional exports, declarations and source maps. Browser Vite/esbuild and mobile Metro/Hermes platform checks passed. RPC proof orchestration is excluded from portable bundles.
• Prior sync candidate TypeScript conformance: 6,483 passed / 211 existing skips; structural validation: 6,694 vectors. No skip or conformance threshold was added.
• pnpm audit:security: existing governed audit policy passed; advisory dispositions unchanged. No new dependency override, quality suppression or finding-baseline waiver.
• Companion desktop: 213 tests, TypeScript and production build passed again with the combined linked candidate; lint and security checks also passed during the earlier follow-up. Current error guidance distinguishes source upgrade requirements, timeouts and unresolved proof verification.

The new authenticated regression checks 250 synthetic proofs, rejects an invalid next page without changing data/checkpoint, then resumes using a validated replacement. Both new and existing destination proof rows are covered. Read-only production diagnostics and local validation of source records supplement these fixtures; no production import or full-backup retest was performed for this follow-up.

Earlier native E2E additionally completed a fresh 134,240-record / 817-page device copy and exercised all six existing profiles with saved local copies. That run took 45m 29s and is completion evidence, not a speedup comparison. A separate bounded native sample measured foreground timer-delay p95 22ms, maximum 44ms. No offline spending or primary-promotion test was performed.

Portable budget changes include measured sync and upstream security costs. The artifact-cost table records macOS measurements and final raw/gzip/Brotli ceilings, including the earlier Linux compression allowance. SDK/browser/mobile composition and artifact checks passed locally; no Node storage backend was added to portable packages.

Dependency evidence

• Release notes and necessity: The package-manifest diff adds the sync benchmark command; it does not add or upgrade a dependency. Sync and migration changes are recorded in the package release notes.
• Runtime, build, and peer compatibility: Existing dependency versions and peer ranges are retained; Node, browser and mobile consumer checks cover the combined source changes.
• Deduplicated lockfile: The committed lockfile is unchanged relative to current upstream main; CI uses frozen installation with lifecycle scripts denied.
• Audit and CodeQL: The governed audit passes with unchanged advisory dispositions. Exact-head CodeQL and zero-new-Sonar checks are required before readiness.
• Package and consumer tests: Full Toolbox tests, focused SDK tests, portable package tests and packed consumer/platform checks are recorded above.
• Bundle and performance impact: The transfer guide records measured platform artifacts and explicit ceilings, including Linux esbuild gzip at 369,222 bytes against a 369,500-byte ceiling. Copy timings and their limits are summarized above.
• Affected public package versions: SDK 2.5.0 and Toolbox/server/client/mobile 2.12.0 candidates; publication is separate from source merge.

Release and operations

• Candidates: SDK 2.5.0, Toolbox/server/client/mobile 2.12.0. Published Toolbox 2.11.0 lacks these transfer methods; version labels alone do not establish capability. Matching client artifacts and support on each relevant source/destination are required. A destination upgrade cannot remove a legacy source response limit.
• Transfers remain bounded to the negotiated 64 MiB frame maximum; larger frames fail without skipping records or advancing checkpoints. This is not unlimited record streaming.
• Providers need additive staging migration 2026-09-09-001 before advertising transfer version 1. syncTransfers: false supports staged rollout. Rollback must preserve current wallet data and reverse only the staging migration using the new runtime before reverting to incompatible older code; see the transfer guide.
• IndexedDB schema 6 preserves data on upgrade; clients requesting schema 5 cannot reopen it. Keep a compatible runtime for retained copies.
• BRC-38/39 export formats and existing BRC-40 ordinary-page contracts remain unchanged; the transfer envelope is optional. No specification revision is proposed here.
• No merge or npm publication was performed. The exact reviewed revision was deployed to a compatible backup provider: health, authenticated identity, checkpoint/transfer capabilities, unchanged configuration/schema and before/after wallet checkpoint preservation checks passed. An isolated authenticated HTTP/SQLite fixture also passed using the staged runtime; it did not open the production database. Source-wallet diagnostics remain read-only. A fresh live full-backup retest remains necessary before claiming the reported scenario resolved end to end.
• All real-wallet and infrastructure evidence stays private. Public fixtures are synthetic; no wallet identities, transaction contents, credentials or personal infrastructure details are included.

[codecov]

Codecov Report

❌ Patch coverage is 93.21534% with 46 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...-toolbox/src/storage/remoting/StorageClientBase.ts	90.69%	0 Missing and 8 partials ⚠️
...llet-toolbox/src/storage/remoting/StorageServer.ts	88.57%	2 Missing and 6 partials ⚠️
...lbox/src/storage/remoting/KnexSyncTransferStore.ts	94.73%	0 Missing and 5 partials ⚠️
...allet-toolbox/src/storage/remoting/SyncTransfer.ts	94.18%	0 Missing and 5 partials ⚠️
...ox/src/storage/remoting/entityValidationHelpers.ts	71.42%	2 Missing and 2 partials ⚠️
...s/wallet/wallet-toolbox/src/storage/StorageKnex.ts	66.66%	1 Missing and 2 partials ⚠️
packages/sdk/src/wallet/ProtoWallet.ts	80.00%	0 Missing and 2 partials ⚠️
...ges/wallet/wallet-toolbox/src/services/Services.ts	86.66%	1 Missing and 1 partial ⚠️
...es/wallet/wallet-toolbox/src/storage/StorageIdb.ts	94.28%	0 Missing and 2 partials ⚠️
...llet/wallet-toolbox/src/storage/StorageProvider.ts	80.00%	0 Missing and 2 partials ⚠️
... and 5 more

📢 Thoughts on this report? Let us know!

[ty-everett]

Approved for release after resolving the post-security-release version collision. Exact head 1cc9c61 targets fresh SDK 2.6.0 and Wallet Toolbox/Client/Mobile 2.13.0 candidates; local health, format, lint, build, and post-build typecheck gates pass. Merge remains contingent on all hosted exact-head checks succeeding.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud