feat(bench): per-arch baselines, matrix/pgo tasks, methodology doc [W0-12]

.gitignore

@@ -88,3 +88,6 @@ api-reference-*.png
 docs/ARCHITECT_REVIEW_*.md
 docs/audits/
 docs/superpowers/
+
+# benchmark matrix output (task bench:matrix)
+.bench/

CHANGELOG.md

@@ -7,6 +7,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+- **Benchmark and perf-gate infrastructure (performance overhaul, wave 0)**:
+  pull requests are now judged by interleaved A/B measurements instead of
+  absolute numbers. No library behaviour changes apart from #901 (see Fixed). The pieces, by PR:
+  - Corpus and behaviour guards: the benchmark corpus manifest and
+    `internal/benchcorpus` loader (#895); behaviour goldens (token, parse,
+    format, lint, scan and transpile hashes) over every corpus and dialect
+    (#915); wall-clock test assertions replaced with ratio and count checks
+    (#899); ClickHouse/Snowflake lambda bodies and `U&` in dialects without
+    `&` fixed so the goldens could be pinned (#901).
+  - Benchmark matrix: tokenizer and parse stages (#913), server shapes with
+    `AllocsPerRun` budgets, a stack-depth test and a frame-size script (#909),
+    high-level, render and transform (#911), transpile with a prefilter
+    differential test (#912), interop benchmarks and a binary-size budget
+    (#908), and lint, security and LSP (#910).
+  - Competitor harness: new pure-Go peers (dolthub/vitess, rqlite/sql,
+    AfterShip/clickhouse-sql-parser, auxten/postgresql-parser), ORM-style
+    queries, lexer, render, normalize, fresh-goroutine, parallel and
+    per-dialect cells with checked-in common-accept sets, and a schema v2
+    results file that keeps every v1 field (#914).
+  - Gate: an interleaved A/B workflow in shadow mode (`GOGC=off`, `-cpu 1`,
+    ABBA rounds, benchstat at alpha 0.01, A/A noise control; `allocs/op`, `B/op`
+    and instruction-count gates) with `task bench:ab` and `task bench:gate`
+    (#916).
+  - Baselines and docs (#917): per-architecture
+    `benchmarks/baselines/{amd64,arm64}.json` holding trend-only ns/op and gated
+    B/op and allocs/op, checked or regenerated with `task bench:baseline` and
+    `task bench:baseline:update` (local-only until W0-13 wires the check into
+    `perf-gate.yml`; CI currently gates allocs/op and B/op relatively via the
+    A/B gate); new `task bench:matrix` and `task pgo:refresh`;
+    `docs/PERFORMANCE_METHODOLOGY.md` (method, how to run an A/B, how the gate
+    decides, claims policy). `performance_baselines.json` and
+    `TestPerformanceRegression` are now documented as a smoke test only, and
+    its never-enforced `TokenizationThroughput` and `EndToEndSustained` entries
+    are removed.
+
 ### Performance
 - **Tokenizer**: keyword and compound-keyword-start lookup now uses a single fused-hash open-addressed table (one FNV-1a hash computed while upper-casing, one probe) instead of two Go map lookups per identifier. The table is filled in a fixed order (hot keywords first, then sorted) so its probe layout is deterministic across processes. No behaviour change; token output is identical [W2-01].
 - **Pools**: the tokenizer, parser and AST pools now count Get/Put through precreated, cache-line padded atomic counters (`internal/poolstats`) instead of an RWMutex-guarded map keyed by pool name, removing a lock, map lookup and string hash from every pool operation. `metrics.GetPoolStats` and `metrics.ResetPoolStats` behave exactly as before; named custom pools keep the map path. [W1-01]

CLAUDE.md

@@ -191,8 +191,7 @@ go test -bench=. -benchmem -cpuprofile=cpu.prof ./pkg/...    # With profiling
 ```
 
 ### Performance Regression Tests
-- Baselines defined in `performance_baselines.json` at project root
-- CI environment variability may require baseline adjustments (tolerance %)
+- `TestPerformanceRegression` is a smoke test only (loose ceilings in `performance_baselines.json`); the real gate is `task bench:gate` plus `task bench:baseline` (`benchmarks/baselines/<arch>.json`), see `docs/PERFORMANCE_METHODOLOGY.md`
 - Run locally: `go test -run TestPerformanceRegression ./pkg/sql/parser/`
 - Skip with race detector (adds 3-5x overhead): automatically skipped
 

Taskfile.yml

@@ -201,6 +201,39 @@ tasks:
       ROUNDS: '{{default "10" .ROUNDS}}'
       BENCHTIME: '{{default "200ms" .BENCHTIME}}'
 
+  bench:matrix:
+    set: [pipefail]
+    desc: "Run the whole benchmark matrix once (every pkg benchmark except cbinding, which needs CGO: see bench:interop; -benchmem, -cpu 1, GOGC=off) and write it to OUT (default: .bench/matrix.txt, gitignored). Vars: BENCHTIME (default 100ms), OUT, FILTER (default .)"
+    env:
+      GOGC: 'off'
+    cmds:
+      - mkdir -p "$(dirname '{{.OUT}}')"
+      - go test -run '^$' -bench '{{.FILTER}}' -benchmem -benchtime {{.BENCHTIME}} -cpu 1 $(go list ./pkg/... ./wasm/internal/... | grep -v /cbinding) 2>&1 | tee {{.OUT}}
+    vars:
+      BENCHTIME: '{{default "100ms" .BENCHTIME}}'
+      FILTER: '{{default "." .FILTER}}'
+      OUT: '{{default ".bench/matrix.txt" .OUT}}'
+
+  bench:baseline:
+    desc: "Check allocs/op and B/op of the gated suite against benchmarks/baselines/<GOARCH>.json (ns/op is trend-only). Fails if allocs/op rose or B/op rose more than 2%"
+    cmds:
+      - go run ./scripts/perf/baseline {{.CLI_ARGS}}
+
+  bench:baseline:update:
+    desc: "Re-measure the gated suite and rewrite benchmarks/baselines/<GOARCH>.json (run on the architecture being recorded, on an idle machine)"
+    cmds:
+      - go run ./scripts/perf/baseline -update {{.CLI_ARGS}}
+
+  pgo:refresh:
+    desc: "Regenerate cmd/gosqlx/default.pgo from a CPU profile of the parse benchmarks (Go applies it automatically to the CLI build). Vars: BENCHTIME (default 10s)"
+    cmds:
+      - go test -run '^$' -bench '^BenchmarkParseE2E$' -benchtime {{.BENCHTIME}} -cpu 4 -cpuprofile cmd/gosqlx/default.pgo -o {{.TMP_BIN}} ./pkg/gosqlx
+      - /bin/rm -f {{.TMP_BIN}}
+      - echo "Wrote cmd/gosqlx/default.pgo. Verify with task bench:ab before committing it."
+    vars:
+      BENCHTIME: '{{default "10s" .BENCHTIME}}'
+      TMP_BIN: '{{.TMPDIR | default "/tmp"}}/gosqlx-pgo.test'
+
   bench:profile:
     desc: "CPU profiles (-cpu 1, GOGC=off, -cpu 4) of the gap queries plus perf stat counters when available (Vars: BENCHTIME, default 3s)"
     cmds:

benchmarks/baselines/amd64.json

@@ -0,0 +1,6 @@
+{
+  "schema": 1,
+  "arch": "amd64",
+  "comment": "Empty on purpose: recorded on amd64 hardware at checkpoint C0 (W0-13) with task bench:baseline:update. The check refuses an empty baseline.",
+  "benchmarks": {}
+}

benchmarks/baselines/arm64.json

@@ -0,0 +1,139 @@
+{
+  "schema": 1,
+  "arch": "arm64",
+  "comment": "ns_per_op is trend-only. bytes_per_op and allocs_per_op are gated by scripts/perf/baseline (allocs/op must not rise, B/op may rise at most 2 percent). Measured with GOGC=off, -cpu 1, medians. Regenerate with: task bench:baseline:update",
+  "updated": "2026-11-01",
+  "go_version": "go1.26.9",
+  "benchmarks": {
+    "pkg/gosqlx:BenchmarkParseE2E/aggregate": {
+      "ns_per_op": 1946,
+      "bytes_per_op": 1856,
+      "allocs_per_op": 18
+    },
+    "pkg/gosqlx:BenchmarkParseE2E/complex": {
+      "ns_per_op": 7593,
+      "bytes_per_op": 6552,
+      "allocs_per_op": 50
+    },
+    "pkg/gosqlx:BenchmarkParseE2E/cte": {
+      "ns_per_op": 2405,
+      "bytes_per_op": 2392,
+      "allocs_per_op": 21
+    },
+    "pkg/gosqlx:BenchmarkParseE2E/delete": {
+      "ns_per_op": 732.6,
+      "bytes_per_op": 400,
+      "allocs_per_op": 5
+    },
+    "pkg/gosqlx:BenchmarkParseE2E/insert": {
+      "ns_per_op": 1328,
+      "bytes_per_op": 824,
+      "allocs_per_op": 17
+    },
+    "pkg/gosqlx:BenchmarkParseE2E/join": {
+      "ns_per_op": 2839,
+      "bytes_per_op": 2360,
+      "allocs_per_op": 17
+    },
+    "pkg/gosqlx:BenchmarkParseE2E/select_where": {
+      "ns_per_op": 1796,
+      "bytes_per_op": 1560,
+      "allocs_per_op": 18
+    },
+    "pkg/gosqlx:BenchmarkParseE2E/simple_select": {
+      "ns_per_op": 708.3,
+      "bytes_per_op": 816,
+      "allocs_per_op": 5
+    },
+    "pkg/gosqlx:BenchmarkParseE2E/subquery": {
+      "ns_per_op": 1666,
+      "bytes_per_op": 1872,
+      "allocs_per_op": 14
+    },
+    "pkg/gosqlx:BenchmarkParseE2E/update": {
+      "ns_per_op": 1151,
+      "bytes_per_op": 736,
+      "allocs_per_op": 13
+    },
+    "pkg/gosqlx:BenchmarkParseE2E/window": {
+      "ns_per_op": 1596,
+      "bytes_per_op": 1312,
+      "allocs_per_op": 11
+    },
+    "pkg/gosqlx:BenchmarkParseFresh/complex": {
+      "ns_per_op": 14302,
+      "bytes_per_op": 6632,
+      "allocs_per_op": 51
+    },
+    "pkg/gosqlx:BenchmarkParseFresh/join": {
+      "ns_per_op": 8903,
+      "bytes_per_op": 2440,
+      "allocs_per_op": 18
+    },
+    "pkg/gosqlx:BenchmarkParseFresh/simple_select": {
+      "ns_per_op": 3877,
+      "bytes_per_op": 896,
+      "allocs_per_op": 6
+    },
+    "pkg/gosqlx:BenchmarkParseRW/clickhouse/medium": {
+      "ns_per_op": 47552,
+      "bytes_per_op": 38024,
+      "allocs_per_op": 384
+    },
+    "pkg/gosqlx:BenchmarkParseRW/mariadb/medium": {
+      "ns_per_op": 73623,
+      "bytes_per_op": 66504,
+      "allocs_per_op": 530
+    },
+    "pkg/gosqlx:BenchmarkParseRW/mysql/medium": {
+      "ns_per_op": 49300,
+      "bytes_per_op": 44816,
+      "allocs_per_op": 368
+    },
+    "pkg/gosqlx:BenchmarkParseRW/oracle/medium": {
+      "ns_per_op": 166723,
+      "bytes_per_op": 116496,
+      "allocs_per_op": 1094
+    },
+    "pkg/gosqlx:BenchmarkParseRW/postgresql/medium": {
+      "ns_per_op": 72910,
+      "bytes_per_op": 72216,
+      "allocs_per_op": 648
+    },
+    "pkg/gosqlx:BenchmarkParseRW/snowflake/medium": {
+      "ns_per_op": 103799,
+      "bytes_per_op": 85416,
+      "allocs_per_op": 779
+    },
+    "pkg/gosqlx:BenchmarkParseRW/sqlite/medium": {
+      "ns_per_op": 39994,
+      "bytes_per_op": 40896,
+      "allocs_per_op": 348
+    },
+    "pkg/gosqlx:BenchmarkParseRW/sqlserver/medium": {
+      "ns_per_op": 138485,
+      "bytes_per_op": 92763,
+      "allocs_per_op": 937
+    },
+    "pkg/gosqlx:BenchmarkPoolGetPut/serial": {
+      "ns_per_op": 65.17,
+      "bytes_per_op": 0,
+      "allocs_per_op": 0
+    },
+    "pkg/sql/tokenizer:BenchmarkTokenize/scratch/complex": {
+      "ns_per_op": 3521,
+      "bytes_per_op": 0,
+      "allocs_per_op": 0
+    },
+    "pkg/sql/tokenizer:BenchmarkTokenize/scratch/insert": {
+      "ns_per_op": 611.7,
+      "bytes_per_op": 0,
+      "allocs_per_op": 0
+    },
+    "pkg/sql/tokenizer:BenchmarkTokenize/scratch/join": {
+      "ns_per_op": 1337,
+      "bytes_per_op": 0,
+      "allocs_per_op": 0
+    }
+  }
+}

docs/PERFORMANCE_METHODOLOGY.md

@@ -0,0 +1,98 @@
+# Performance Methodology
+
+How GoSQLX measures performance, how a pull request is judged, and what we may claim in public. Commands assume the repository root and the [Task](https://taskfile.dev) runner.
+
+## Principles
+
+1. **A/B, never absolute.** A change is judged by a base build and a head build measured in the same job on the same machine. Absolute ns/op numbers from different machines or days are not comparable.
+2. **Deterministic metrics gate; time gates only above the noise floor.** `allocs/op` and `B/op` do not depend on machine load, so they are hard gates. Time is a gate only for a statistically significant, material change.
+3. **Report relative deltas.** Publish "-12.4% (p=0.002, n=10)", not a bare nanosecond figure.
+4. **Behaviour is guarded too.** The golden hashes (`testdata/goldens`) must be unchanged; a speedup that changes output is a bug.
+
+## The benchmark matrix
+
+Most functionality-by-dialect cells have a benchmark; lint, security and LSP are pending (#910). List them with `go test -list Benchmark ./pkg/...`. Run the whole matrix once with:
+
+```bash
+task bench:matrix                      # all pkg benchmarks, -cpu 1, GOGC=off -> .bench/matrix.txt
+task bench:matrix FILTER='^BenchmarkParseRW$' BENCHTIME=200ms
+```
+
+`pkg/cbinding` needs CGO and is run by `task bench:interop`. Competitor parsers are measured separately by `task bench:compare` (single pass) and `task bench:publish` (`-count=10`, writes `benchmarks/compare/results/latest.{json,txt}`).
+
+## Running an A/B comparison
+
+```bash
+task bench:ab                          # working tree vs origin/main, gated suite
+task bench:ab BASE=v1.17.0 ROUNDS=15
+task bench:ab BENCH='./pkg/gosqlx=^BenchmarkParseE2E$'    # one benchmark
+task bench:gate                        # exactly what CI runs, locally
+```
+
+`scripts/perf/ab.sh` builds test binaries for both revisions with the same toolchain and runs them interleaved in ABBA order for at least 10 rounds. The primary metric uses `GOGC=off GOMAXPROCS=1` (`-cpu 1`, pinned to one CPU where `taskset` exists): garbage-collection-free mutator time has low variance and ranks queries the same way the `-cpu 4` headline does. A secondary default-GC `-cpu 1` arm (`-g`, enabled by `bench:gate`) is reported for information only; `-cpu 4` is measured only by `task bench:profile` and the competitor harness. Results go through `benchstat` at alpha 0.01. The benchmarks that are gated are listed in `scripts/perf/gate-suite.txt`.
+
+Tips: run on an idle machine, close other benchmarks, and re-run once if the A/A control (below) reports noise. Never weaken a test or a threshold to make a result pass.
+
+## How the gate decides
+
+`scripts/perf/gate` evaluates the run (rules from PLAN section 2.6). A PR fails when any of these holds:
+
+| Check | Rule |
+|---|---|
+| Time | a gated benchmark is 3% or more slower at p < 0.01 (Mann-Whitney U), or the gated geomean is 1.5% or more slower |
+| `allocs/op` | any gated benchmark allocates more |
+| `B/op` | any gated benchmark rises by more than 2% |
+| Instructions/op | any case executes more than 1.5% more instructions (`cmd/perfdriver`) |
+
+**A/A control.** Each run also compares the base binary against a second copy of itself. If more than 10% of benchmarks move by more than 3% in that A/A comparison, the machine is too noisy: the job re-runs once, and a second noisy result is **inconclusive**, which is neutral, not a failure.
+
+**Verdicts:** `pass`, `regression`, `noisy` (re-run), `inconclusive`. The gate starts in shadow mode (it reports but never fails); enforcement is switched on after the shadow period. A maintainer can accept a known regression with the `perf-regression-accepted` label plus a written justification in the PR body.
+
+## Baseline files
+
+`benchmarks/baselines/<goarch>.json` (`amd64.json`, `arm64.json`) records, for each gated benchmark, `ns_per_op` (trend only, never gated), `bytes_per_op` and `allocs_per_op` (gated). Keys are `<package path>:<benchmark name>`.
+
+```bash
+task bench:baseline                    # re-measure, fail if allocs/op rose or B/op rose >2%
+task bench:baseline:update             # re-measure and rewrite benchmarks/baselines/<GOARCH>.json
+go run ./scripts/perf/baseline -input out.txt     # check recorded `go test -bench -benchmem` output
+```
+
+`task bench:baseline` is local-only: no workflow runs `scripts/perf/baseline`, and it cannot run on CI's amd64 until `amd64.json` is recorded and the check is wired into `perf-gate.yml` (W0-13). In CI, allocs/op and B/op are currently gated relatively by the A/B gate. When recording from external output use `-update -input out.txt -go-version <toolchain>`; the local toolchain is never stamped onto external numbers.
+
+Update a baseline only in a PR whose purpose or measured effect justifies it, and say why in the PR body. Record each architecture on that architecture's hardware; an empty file is refused by the check instead of passing silently. `amd64.json` is recorded at the first amd64 checkpoint.
+
+The check fails (it does not pass silently) when a full measurement leaves a baseline entry unmeasured or finds a benchmark the baseline lacks, because a suite-regex change rekeys entries: run `task bench:baseline:update`. It also fails when no baseline benchmark was measured, and with `-input` it rejects output whose `goarch:` header differs from `-arch`. allocs/op and B/op can shift between Go minors, so the check warns when the baseline's `go_version` differs in major.minor from the running toolchain; record baselines with the toolchain CI uses.
+
+Deviation from PLAN 2.7: the baseline files do not store instructions/op (Ir). Ir is gated relatively by `perfdriver` in the A/B gate instead.
+
+`performance_baselines.json` and `TestPerformanceRegression` (`pkg/sql/parser`) remain only as a **smoke test**: loose ns/op ceilings with 60-65% tolerances that catch order-of-magnitude breakage on noisy CI. They are not a gate and their tolerances must not be tightened to act as one.
+
+## Profiling and PGO
+
+```bash
+task bench:profile                     # pprof at -cpu 1 / GOGC=off / -cpu 4 for the gap queries
+task pgo:refresh                       # regenerate cmd/gosqlx/default.pgo from the parse benchmarks
+```
+
+A refreshed PGO profile is committed only after `task bench:ab` shows a gain and no gated regression.
+
+## Claims policy
+
+Public performance statements (README, website, release notes) follow these rules:
+
+- **Form:** "fastest of N measured on `<corpus, dialect, arch>`". N, the corpus, the dialect and the architecture must all be named.
+- **Measurement:** same machine, interleaved, in-process, with raw JSON published and a one-command reproduction (`task bench:publish`).
+- **Corpus:** the headline corpus is the set of statements every listed parser accepts. Acceptance rate is reported next to speed.
+- **Ties:** a gap under 3% is a tie.
+- **Per-cell evidence:** a claim needs every relevant peer measured in that cell. No Go peer in a dialect means only "the only Go parser with X", not a speed claim.
+- **Rust and C:** no claim against Rust or C parsers until the cross-language harness has run on the same machine and corpus; then report the result whatever it is.
+- **No peer:** for lint, security scan, transform, LSP, bindings and WASM we publish latency budgets only, never "fastest".
+- **Timing:** website claims ship with the release docs PR, not on code merge.
+
+## Reference
+
+- `scripts/perf/ab.sh`, `scripts/perf/gate/`, `scripts/perf/gate-suite.txt`: A/B runner, verdict logic, gated suite
+- `scripts/perf/baseline/`: baseline recorder and checker
+- `.github/workflows/perf-gate.yml`: CI gate; `.github/workflows/bench-compare.yml`: competitor runs
+- `benchmarks/compare/README.md`: competitor harness