test: add proxy burn-in tooling

[freshtonic]

Summary

• add a dedicated cipherstash-proxy-burn-in workspace package with deterministic conformance coverage
• add a timed soak workload with configurable duration, concurrent CRUD, release-profile Proxy build and launch, and one-second RSS sampling
• adapt the pg-proto type-lab and commerce fixtures into uniquely named public tables with representative EQL v3 integer, text, and JSON domains
• install EQL when needed, apply fixture DDL through Proxy, and seed encrypted values on a fresh connection after schema and encrypt-config reload
• use unqualified fixture names so EQL Mapper resolves every workload statement
• fail conformance unless direct PostgreSQL reads prove ciphertext is stored at rest, then verify typed plaintext is returned through Proxy
• run the encrypted burn-in in a dedicated PostgreSQL 17 CI job and upload its RSS report
• document credentials, EQL setup, module responsibilities, encryption-path invariants, conformance runs, soak runs, and generated reports

Soak reliability

• discover and execute Cargo's exact release artifact, configure it from the direct database target, and reject occupied listeners
• retain and monitor the owned child process so an unrelated Proxy cannot satisfy readiness
• bound readiness, operations, and worker shutdown; terminate the child on interruption or drop
• redact connection credentials from help, diagnostics, and reports
• serialize fixture mutation with a PostgreSQL advisory lock
• write reports atomically with terminal status, errors, elapsed time, artifact provenance, operation counts, and non-zero RSS evidence

Verification

• cargo fmt --all -- --check
• RUSTC_WRAPPER= cargo test -p cipherstash-proxy-burn-in
• RUSTC_WRAPPER= cargo clippy -p cipherstash-proxy-burn-in --all-targets -- -D warnings
• conformance run against the encrypted fixtures
• isolated two-second release soak using a custom CARGO_TARGET_DIR: 15 encrypted CRUD cycles, zero errors, live RSS report generated
• occupied-listener regression: soak failed before spawning instead of attaching to the existing listener
• GitHub reports all six commits as validly signed with james@cipherstash.com and one DCO sign-off each

[tobyhede]

Proxy already has benchmark setup that might be worth extending if it doesn't do what you need

tests/benchmark/ (in main, CI-wired via mise run benchmark/benchmark:continuous) — pgbench-driven, black-box, comparative: Proxy vs raw Postgres vs pgbouncer vs pgcat, plaintext vs encrypted.
Plots results to PNG/CSV, CI handles as regression
Protocol-level throughput/latency comparison.

[tobyhede]

Review: Standards & Spec (vs origin/main, single commit 179b9dc)

Two-axis review — does the code follow the repo's documented standards, and does it match what the PR description asked for — with every finding independently re-verified against the actual code before reporting.

Worth fixing

1. benches/proxy_crud.rs reimplements what cipherstash-proxy-burn-in already provides

proxy_crud.rs:6-7 re-includes the migration SQL via a relative path:

const SCHEMA: &str = include_str!("../../cipherstash-proxy-burn-in/migrations/0001_schema.sql");
const SEED: &str = include_str!("../../cipherstash-proxy-burn-in/migrations/0002_seed.sql");

but cipherstash-proxy-burn-in::lib already exports these as pub const SCHEMA_MIGRATION / SEED_MIGRATION. Same story for connect() (near-duplicate of database::connect), the CRUD shape (realistic_crud mirrors soak::crud_cycle's insert → read → update → cascading-delete skeleton), and the connection-string defaults (duplicated verbatim between main.rs and proxy_crud.rs).

Adding cipherstash-proxy-burn-in as a dev-dependency of cipherstash-proxy would fix all of these at once — no dependency cycle results, and nearly all of burn-in's deps (clap, serde, tokio, tokio-postgres) are already direct deps of cipherstash-proxy, so it shouldn't meaningfully affect bench build time.

2. README doesn't document the credentials the release Proxy binary needs

The soak workload spawns a real release-profile Proxy binary that inherits the parent process's environment, but the README only says credentials "must already be available in the environment" — no variable names, no pointer to mise.local.toml. packages/showcase/README.md:438-445 sets a good precedent here (lists CS_WORKSPACE_CRN, CS_CLIENT_ACCESS_KEY, CS_DEFAULT_KEYSET_ID, CS_CLIENT_ID, CS_CLIENT_KEY explicitly) — worth matching that so a new contributor can actually get soak running from the README alone.

Minor, not blocking

• conformance.rs repeats the literal 900_001_i32 nine times rather than binding it once (the repo's convention elsewhere — e.g. random_id() in the integration suite — binds once and reuses).

Checked and cleared (no action needed)

• The RSS sampler's first tick landing at t≈0 rather than t≈1s is expected tokio::time::interval behavior, and it's actually useful here — it becomes initial_rss_bytes, a genuine baseline reading.
• --max-rss-growth-mib isn't called out in the PR description's bullets, but it's a natural, opt-in extension for a burn-in/soak tool (the whole point is catching leaks) — not scope creep worth flagging.
• The migrations' "copied from pg-proto" provenance checks out — pg-proto is a real repo by the same author, already a workspace dependency of this codebase.

[freshtonic]

Proxy already has benchmark setup that might be worth extending if it doesn't do what you need

Ah, I just looked for a benches dir and missed that. I'll remove mine.

[freshtonic]

Addressed in f517e6db (with the duplicate CRUD benchmark already removed in af008d7b):

• documented CS_WORKSPACE_CRN, CS_CLIENT_ACCESS_KEY, CS_DEFAULT_KEYSET_ID, CS_CLIENT_ID, and CS_CLIENT_KEY in the burn-in README, with mise.local.toml setup
• removed the remaining stale Criterion benchmark documentation
• bound the conformance fixture ID once and reused it

Verified with formatting, the burn-in package tests, and Clippy with warnings denied.

[tobyhede]

Review: Correctness at ff0243ba

The two follow-up commits fix the original aggregate type mismatch, add an automated encrypted soak, and verify ciphertext at rest. I rechecked the remaining findings against the new head; these still need attention.

Blocking

1. Release artifact discovery still ignores Cargo configuration — src/soak.rs:223-253 [repro]

cargo build respects CARGO_TARGET_DIR, build.target-dir, and configured build targets, but spawn_release_proxy() always opens workspace/target/release/cipherstash-proxy. On a shared-target setup, the run either fails after building or launches a stale workspace binary.

Consume Cargo’s JSON compiler-artifact.executable instead of reconstructing the path. That identifies the artifact produced by this exact build and handles target triples as well as custom target directories.

2. An existing listener on 6432 still makes soak measure a dead process — src/soak.rs:63-116, src/database.rs:183-193 [repro]

The child is reduced to a PID before readiness and is never checked again. With another Proxy already listening, the newly built child exits with Address already in use, readiness and fixture setup use the existing Proxy, and RSS sampling targets the dead child. I reproduced a successful run with 1,701 CRUD cycles, zero final RSS, and zero reported growth.

Preflight the configured bind address, but do not treat that as proof of identity—it remains racy. Pass the Child into the run loop, check try_wait() throughout startup, migration, sampling, and worker completion, and reject zero RSS samples.

3. A timed soak can run indefinitely and runtime failures can erase its report — src/soak.rs:89-136, src/database.rs:183-193

Connection, startup-handshake, readiness-query, and CRUD futures have no deadlines. Once the sampling deadline passes, join_next() can wait forever for a wedged worker.

Sampling and worker errors also propagate before the report is written. Add bounded readiness attempts, per-operation timeouts, and a bounded worker-shutdown period. Preserve partial samples and record the terminal error before returning failure.

4. Database credentials are exposed in help and errors — src/main.rs:31-50, src/database.rs:17-20,183-190 [repro]

Clap prints the complete BURN_IN_*_DATABASE_URL environment value in --help, and connection/readiness errors interpolate the same URL.

Use hide_env_values = true and a parsed connection type with a redacted display form. Never include raw database URLs in diagnostics or reports.

Should fix

5. The aggregate assertion still panics on its intended failure case — src/conformance.rs:86-97

The new ::bigint cast fixes the unconditional numeric decoding panic. However, if the join loses every row, sum(...)::bigint is NULL and get::<_, i64>() still panics before "joined CRUD result was corrupted" can fire. Decode with try_get::<_, Option<i64>>() and assert Some(4_998).

6. Concurrent runs can deadlock or invalidate one another — migrations/0001_schema.sql, migrations/0002_seed.sql:3-5, src/soak.rs:156-219

Every run drops and recreates the public fixture tables. The remaining TRUNCATE order is also the reverse of the CRUD insert order. Reordering prevents that specific lock inversion, but concurrent runs would still destroy or contaminate each other’s fixtures and measurements.

Acquire a run-level advisory lock and retain its connection for the entire conformance or soak run; a migration-only lock is insufficient.

7. Report and gate semantics remain inconsistent — src/soak.rs:113-153

A worker error already short-circuits at join_next(), so ensure!(report.errors == 0) cannot observe one. Zero completed cycles can pass, output-path errors are discovered only after the workload, and soak passed prints before the RSS gate—I reproduced it printing success immediately before exiting 1.

Require at least one completed cycle, include terminal status in partial reports, preflight output with a sibling temporary file and atomic rename, and print success only after all gates pass.

8. The migrated database may not be the spawned Proxy’s upstream — src/soak.rs:61-74, src/soak.rs:247-253

--direct-database-url selects the database where EQL is installed and ciphertext is inspected, but the child Proxy is spawned without database arguments and reads ambient CS_DATABASE__*. An override can therefore migrate one database while the child serves another.

Pass or validate the child’s upstream configuration. Record sanitized provenance in the report: artifact hash or commit, concurrency, timestamp, actual elapsed duration, and a redacted database identity.

Additional notes

• The new CI job closes the encryption-path and execution gaps. It runs only soak and omits --max-rss-growth-mib, so deterministic conformance and retained-growth gating remain local-only. Adding them would strengthen coverage once a stable threshold is established.
• First-to-last RSS delta matches the README’s “retained growth” wording. The immediate first tick is still a cold baseline; add a warm-up or delayed first sample. Trend fitting is optional rather than a correctness requirement.
• Add kill_on_drop(true) and signal handling so panics and interrupts do not leave an owned child or discard all samples.
• Make the wide-text assertion exact, use checked multiplication for --max-rss-growth-mib, and establish a fixture migration/version strategy before the schema evolves again.
• The updated package’s three unit tests pass. The live subcommands remain the only tests of lifecycle, encryption, and report behavior.

Separately, commits 5c72f021, 31836df6, d64fcba4, and ff0243ba lack the repository-required DCO sign-off, and src/main.rs:61 should say “connections,” not “sessions.”

[freshtonic]

Addressed the latest review in c9dbd6f9:

• Cargo JSON now supplies the exact release executable; custom target directories are supported.
• The soak rejects occupied listeners, retains its spawned child, checks try_wait() throughout, and rejects zero RSS samples.
• Readiness, individual operations, migration, and worker shutdown are bounded; interruption/drop terminates the child.
• Reports are written atomically and include terminal status/error, actual elapsed time, artifact/source/database provenance, operations, errors, and partial RSS evidence.
• Database arguments are parsed into a credential-redacting type, hidden from environment-backed help text, and the spawned Proxy is configured from the direct target.
• A run-wide PostgreSQL advisory lock prevents concurrent fixture mutation; truncation follows dependency order.
• The aggregate assertion handles SQL NULL explicitly, the wide-text assertion is exact, RSS conversion is checked, and CLI wording says connections.

Regression coverage includes credential redaction, custom Cargo artifact discovery, report validity, occupied-listener rejection, and an end-to-end custom-target soak. The burn-in package tests and Clippy pass. I also rewrote the stack: GitHub reports every commit as validly signed by james@cipherstash.com, with exactly one matching DCO sign-off.

[tobyhede]

Review: burn-in tooling at c5cc4f44

I re-verified every finding against the code and against completed CI runs. Two candidate
findings did not hold and are listed under "Checked and cleared".

Should fix

1. The burn-in builds Proxy in release mode two times — .github/workflows/test.yml:93-97, packages/cipherstash-proxy-burn-in/src/soak.rs:306-326

mise run proxy:up calls build:binary, which builds with --target x86_64-unknown-linux-gnu
(mise.toml:680). build_release_proxy() builds the same package with no --target. The two
builds write to different directories and share no artifacts.

Each build takes about 3 minutes 45 seconds. The release profile sets codegen-units = 1
(Cargo.toml:35-37). The cache does not help: a warm run of the same build took 3 minutes 39
seconds. Earlier burn-in runs took 4 to 6 minutes, and the new conformance step has not yet run in
CI. With that step the job needs about 10 minutes of the 15-minute budget.

Pass the same --target in build_release_proxy(), or start the soak from the binary that
build:binary already produced. Do not increase timeout-minutes.

2. schema_changed is a write-once latch — packages/cipherstash-proxy/src/postgresql/context/mod.rs:565-575

set_schema_changed() only writes true. No code writes false. After a connection sends any DDL,
reload_schema_if_changed() therefore becomes an unconditional reload for the rest of that
connection's life.

Each reload sends ReloadCommand::DatabaseSchema and awaits the response (context/mod.rs:771-789).
The handler reloads the schema and the encrypt config (proxy/mod.rs:116-120). One global task
serves all connections, so these reloads become serial across connections.

This defect is already on main: the same unguarded reload runs for non-passthrough connections
(main's backend.rs:283-285) and for Code::Sync (frontend.rs:303). This PR adds one more case.
A psql session against a database with no encrypted columns now pays a schema reload and an
encrypt-config reload on every statement after its first DDL.

Do not revert backend.rs:175-181; that block fixes a real problem. Clear the flag after a
successful reload. An AtomicBool::swap(false, ...) also removes the read-then-reload race.

3. The burn-in CI job cannot reach the new passthrough branch — .github/workflows/test.yml:88-90

postgres:setup applies tests/sql/schema.sql, which creates EQL-domain columns
(tests/sql/schema.sql:38-62). Proxy infers the encrypt config from the schema
(proxy/encrypt_config/manager.rs:88-91). Each connection snapshots that config when it opens, and
is_passthrough() reads the snapshot (context/mod.rs:798-800). The soak Proxy therefore starts
with a non-empty config, and backend.rs:179-181 never runs in that job.

The unit test passthrough_reloads_changed_schema_on_ready_for_query (backend.rs:921-948) does
cover the branch. No test covers the full path from a bare database. The comment at
database.rs:176-178 states the DDL round trip works "including when this database had no encrypted
columns at boot", and nothing proves that end to end.

Run one burn-in against a database that has no encrypted columns at boot.

Nits

• soak.rs:141-146 gives the whole bootstrap the 10-second per-operation budget, but
  conformance.rs:22 gives migrate no timeout. Measured migration time in CI is about 300
  milliseconds, so the budget is safe today. Make the two paths consistent and give the migration its
  own named constant.
• tests/sql/eql-domains-uninstall.sql:14 drops each domain without IF EXISTS. All 52
  public.eql_v3_* domains are AS jsonb, so CASCADE cannot reach a sibling domain and the loop
  cannot fail today. If it ever fails, the DO block rolls back every drop, and psql exits 0
  because no task sets ON_ERROR_STOP. The teardown would then leave stale domains and report
  success. Add IF EXISTS, and set ON_ERROR_STOP=1 on the teardown task.
• On the worker-shutdown-timeout path, soak.rs:213 returns before soak.rs:221-222 refreshes the
  counters. The report can show operation and error counts that are up to 16 seconds old.
  refresh_rss_summary() solves this for RSS; the counters have no equivalent.
• Item 7 of my earlier review is still open. ensure!(report.errors == 0) (soak.rs:414) cannot
  observe an error: both increments are followed by return Err(...) (soak.rs:173-174,
  soak.rs:180-181), and the error propagates at soak.rs:223 before validate_report runs. The
  errors field itself is useful, because the sampling loop copies it into the report and the report
  is written on the failure path. Keep the field. The assertion is harmless as an invariant guard, so
  no action is needed unless you make worker errors non-fatal.

Checked and cleared

• conformance.rs:125 expect_err(...): the message gives meaningful context, which matches
  CLAUDE.md. The exit status and the skipped checks are the same as with an anyhow error.
• ids cannot overflow i32. It starts at 1,000,000, and i32::try_from returns an error instead of
  wrapping. The limit is about 2.1e9 cycles.
• Sampling and worker deadlines are correctly ordered. A final CRUD cycle can run up to
  OPERATION_TIMEOUT (10 s) past the deadline, and WORKER_SHUTDOWN_TIMEOUT is 15 s.
• No report is written when the run fails before soak.rs:88. preflight_output() and
  if-no-files-found: warn handle this deliberately.
• The ZeroKMS and CTS handshake runs at Proxy startup (proxy/mod.rs:58-62), so READY_TIMEOUT
  covers it, not the migration timeout. Measured init time was 667 ms.
• find_proxy_artifact, the atomic report write, kill_on_drop plus run_until_interrupted, and
  the RSS growth and peak helpers all read correctly.

[freshtonic]

Addressed in bc94d74e:

• Replaced the write-once schema_changed lock with an AtomicBool dirty flag. Both simple and extended query completion now consume the flag through reload_schema_if_changed(), and a failed reload restores it for retry. The async regression test proves two checks after one DDL emit exactly one reload.
• Changed the CI burn-in setup to start from a guaranteed bare database: download EQL, run strict idempotent teardown, then let the soak install EQL before its owned Proxy starts and create encrypted fixture columns through that initially-passthrough Proxy.
• Introduced one named migration timeout and applied it to both conformance and soak fixture migration.
• Cancel and drain timed-out workers before snapshotting operation/error counters, so the failure report contains the terminal counts rather than the previous sampling tick.
• Added ON_ERROR_STOP=1 to EQL teardown/setup cleanup commands so a failed drop cannot report success.

Two observations did not require code changes:

• The actual PR merge ref does not build Proxy twice. This job calls postgres:up, not proxy:up; postgres:up only starts PostgreSQL. The soak's Cargo JSON build is the sole release Proxy build.
• The authoritative EQL 3.0.4 uninstall script already uses DROP SCHEMA IF EXISTS ... CASCADE for both EQL schemas; there is no checked-in tests/sql/eql-domains-uninstall.sql or non-idempotent 52-domain loop in this repository. The actionable teardown issue was the missing ON_ERROR_STOP, fixed above.

Verification: 133 Proxy unit tests, 8 burn-in tests plus its binary/doc tests, formatting, and Clippy with warnings denied all pass. The commit is SSH-signed and DCO-signed as james@cipherstash.com.