Skip to content

perf(tokenizer): build tokens in place instead of copying them by value [W2-10] - #926

Merged
ajitpratap0 merged 3 commits into
mainfrom
perf/w2-10-tokens-in-place
Nov 2, 2026
Merged

ajitpratap0 merged 3 commits into
mainfrom
perf/w2-10-tokens-in-place

Conversation

@ajitpratap0

@ajitpratap0 ajitpratap0 commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

Summary

New task W2-10 from checkpoint C0. On amd64 the lines that move tokens by value were about 39% of lexer time; on M4 they were about 14%. Those lines are the nextToken/readIdentifier returns of the 40 B models.Token, the build of the 72 B models.TokenWithSpan, and the append. In the amd64 assembly the returned token is spilled with 8-byte stores and then reloaded with 16-byte MOVUPS loads. x86 cannot store-forward those loads, so they stall, and that happens 2-3 times per token.

This PR builds each token in place instead:

  • tokenize grows the output slice first (slices.Grow, which uses the same growth sizes as append), takes &tokens[n] and passes &tw.Token to nextToken. Start and End are then written straight into the slot.
  • nextToken, readIdentifier, readPunctuation, readNumber/finishNumber/invalidNumeric, readQuotedIdentifier, readBacktickIdentifier, readQuotedString and readTripleQuotedString now take out *models.Token and return only error. Success returns go through setToken(out, models.Token{...}). It inlines, and the compiler builds the literal directly in *out (checked in GOARCH=amd64 -S: no stack temp and no MOVUPS reload).
  • The slow reference readers (read*Slow) and the rare prefixed-string and U& readers keep their by-value signatures, so the fast-vs-slow differential tests are unchanged. They are bridged with storeToken, which copies field by field because a whole-struct copy would bring the stalling reload back.
  • If a reader fails, its slot is zeroed and dropped, so the tokBuf[len:cap]-is-zero invariant still holds.

Mechanical size: tokenizer.go has about 130 return sites rewritten (return models.Token{...}, nil became return setToken(out, models.Token{...}), and return models.Token{}, err became return err). The new code is token_out.go plus the loop change.

Test Plan

  • After the merge of origin/main (W2-03 conflict in readPunctuation resolved: the zero-copy @param/@@var path writes through setToken), all gates were re-run on 9535b597.
  • task check: pass (fmt, vet, lint, test:race; W0-02 goldens unchanged in internal/goldens)
  • task test:cbinding: pass
  • go test -count=1 ./pkg/compatibility/...: pass
  • go run honnef.co/go/tools/cmd/staticcheck@latest ./pkg/sql/tokenizer/: clean
  • New tests in pkg/sql/tokenizer/token_out_test.go:
    • TestStoreToken_CopiesEveryField uses reflection, so a field added to models.Token cannot be silently dropped by the bridge.
    • TestTokenize_GrowsPastEstimate: {} pairs make prescan under-estimate the token count, which exercises slot growth on both the public and scratch paths, with exact types, values and spans checked.
    • TestTokenize_ErrorLeavesSpareSlotsZero: 11 error cases, one or more per reader family (quoted, backtick, bracket, U&, E'', N'', triple-quoted, invalid number, unexpected character). Each checks scratch/public error parity, that tokBuf[len:cap] is zero, and that the next call on the same tokenizer matches the public output.

Local A/B (Apple M4, shared machine so indicative only): scripts/perf/ab.sh -r 10 -t 200ms -b './pkg/sql/tokenizer=^BenchmarkTokenize$/^scratch$' -b './pkg/gosqlx=^BenchmarkParseE2E$' origin/main, GOGC=off, -cpu 1, ABBA. Results (first push, base f03332f7): 22 of 22 faster, B/op and allocs/op unchanged.

pkg: github.com/ajitpratap0/GoSQLX/pkg/gosqlx
                       │   sec/op    │   sec/op     vs base                │
ParseE2E/simple_select   513.6n ± 1%   479.4n ± 5%   -6.67% (p=0.002 n=10)
ParseE2E/select_where    1.286µ ± 1%   1.207µ ± 1%   -6.15% (p=0.001 n=10)
ParseE2E/join            1.935µ ± 2%   1.790µ ± 2%   -7.49% (p=0.000 n=10)
ParseE2E/aggregate       1.411µ ± 2%   1.322µ ± 2%   -6.34% (p=0.001 n=10)
ParseE2E/subquery        1.221µ ± 3%   1.149µ ± 2%   -5.90% (p=0.000 n=10)
ParseE2E/cte             1.727µ ± 3%   1.631µ ± 1%   -5.56% (p=0.000 n=10)
ParseE2E/window          1.112µ ± 3%   1.021µ ± 1%   -8.18% (p=0.000 n=10)
ParseE2E/complex         5.274µ ± 1%   4.837µ ± 1%   -8.29% (p=0.000 n=10)
ParseE2E/insert          861.8n ± 0%   766.8n ± 1%  -11.02% (p=0.000 n=10)
ParseE2E/update          750.0n ± 1%   690.5n ± 1%   -7.93% (p=0.000 n=10)
ParseE2E/delete          485.4n ± 1%   446.3n ± 1%   -8.07% (p=0.000 n=10)
geomean                  1.193µ        1.105µ        -7.43%
                               │   sec/op    │   sec/op     vs base                │
Tokenize/scratch/simple_select   184.0n ± 3%   159.5n ± 2%  -13.32% (p=0.000 n=10)
Tokenize/scratch/select_where    546.6n ± 2%   473.1n ± 2%  -13.43% (p=0.000 n=10)
Tokenize/scratch/join            865.9n ± 1%   744.6n ± 2%  -14.01% (p=0.000 n=10)
Tokenize/scratch/aggregate       535.8n ± 1%   444.7n ± 1%  -16.99% (p=0.000 n=10)
Tokenize/scratch/subquery        448.1n ± 1%   397.9n ± 2%  -11.22% (p=0.000 n=10)
Tokenize/scratch/cte             672.3n ± 1%   590.4n ± 3%  -12.19% (p=0.000 n=10)
Tokenize/scratch/window          509.8n ± 1%   445.0n ± 1%  -12.71% (p=0.000 n=10)
Tokenize/scratch/complex         2.363µ ± 1%   2.008µ ± 1%  -15.01% (p=0.000 n=10)
Tokenize/scratch/insert          428.0n ± 1%   350.3n ± 1%  -18.14% (p=0.000 n=10)
Tokenize/scratch/update          348.5n ± 1%   292.9n ± 2%  -15.95% (p=0.000 n=10)
Tokenize/scratch/delete          243.4n ± 1%   214.9n ± 2%  -11.69% (p=0.000 n=10)
geomean                          510.6n        438.6n       -14.09%

After merging origin/main (afec16a4, which adds W2-03 and W3-02) the same local A/B gave the same direction (Tokenize/scratch −8.8% to −18.4%, ParseE2E geomean −7.0%), but other jobs on the machine spread the samples by ±20% to ±270%, so ParseE2E did not reach p<0.01 locally. The amd64 gate below is the authoritative measurement.

amd64 CI perf gate (authoritative), on the post-merge head against current main afec16a4: PASS, 30 of 32 gated benchmarks faster, no regression. AMD EPYC 9V74, Go 1.26.1, GOGC=off, -cpu 1, taskset-pinned, 10 ABBA rounds. A/A control: 0 of 32 benchmarks moved more than 3%.

  • Tokenize/scratch (join/complex/insert): −44.1% / −45.8% / −42.1% (p<0.001). The C0 projection was −8% to −12% for the lexer, and the task threshold is ≥3%.
  • ParseE2E: −4.3% (simple_select) to −20.0% (complex). join −16.3%, aggregate −13.8%, insert −18.1%. The C0 projection was −3% to −5%.
  • ParseRW/*/medium −12.2% to −18.3%, Normalize −4.6% to −13.7%, TransformParse −5.5% to −18.9%, ParseFresh join/complex −7.8%/−11.4%. Suite geomean −16.1%.
  • allocs/op unchanged everywhere. B/op unchanged or lower (down to −0.02%). Instructions/op −3.3% to −5.4% (cachegrind).
  • Secondary metric (default GC, informational): geomean −12.8%.
  • The first push (base f03332f7, before W2-03/W3-02) gave the same picture: suite geomean −15.9%, scratch tokenize −42% to −46%. W3-02 did not absorb this gain.
  • The amd64 lexer gain is about 3x the M4 gain. That fits C0's finding that amd64 amplifies the cost of by-value token copies.
benchmark time p A/A B/op allocs/op status
ParseE2E/simple_select -4.33% <0.001 +0.07% +0.00% 5 → 5 faster
ParseE2E/select_where -11.25% <0.001 -0.19% +0.00% 18 → 18 faster
ParseE2E/join -16.32% <0.001 +0.21% +0.00% 17 → 17 faster
ParseE2E/aggregate -13.83% <0.001 +0.03% +0.00% 18 → 18 faster
ParseE2E/subquery -11.15% <0.001 +0.34% +0.00% 14 → 14 faster
ParseE2E/cte -13.22% <0.001 -0.19% +0.00% 21 → 21 faster
ParseE2E/window -13.36% <0.001 +0.36% +0.00% 11 → 11 faster
ParseE2E/complex -19.95% <0.001 -0.05% -0.02% 50 → 50 faster
ParseE2E/insert -18.13% <0.001 +0.05% +0.00% 17 → 17 faster
ParseE2E/update -13.06% <0.001 -0.34% +0.00% 13 → 13 faster
ParseE2E/delete -10.92% <0.001 -0.05% +0.00% 5 → 5 faster
ParseRW/postgresql/medium -12.22% <0.001 -0.15% -0.01% 646 → 646 faster
ParseRW/mysql/medium -15.40% <0.001 -0.19% -0.00% 365 → 365 faster
ParseRW/mariadb/medium -16.57% <0.001 +0.12% -0.01% 523 → 523 faster
ParseRW/sqlserver/medium -15.91% <0.001 -0.24% -0.01% 928 → 928 faster
ParseRW/oracle/medium -14.72% <0.001 -0.33% -0.00% 1093 → 1093 faster
ParseRW/sqlite/medium -14.40% <0.001 -0.46% +0.00% 346 → 346 faster
ParseRW/snowflake/medium -18.25% <0.001 -0.47% -0.00% 777 → 777 faster
ParseRW/clickhouse/medium -16.81% <0.001 -0.10% +0.00% 384 → 384 faster
ParseFresh/simple_select +0.11% 0.151 -0.02% +0.00% 6 → 6 ok
ParseFresh/join -7.81% <0.001 -0.20% +0.00% 18 → 18 faster
ParseFresh/complex -11.35% <0.001 -0.01% +0.00% 51 → 51 faster
PoolGetPut/serial +0.31% 0.519 +0.09% +0.00% 0 → 0 ok
Tokenize/scratch/join -44.06% <0.001 -0.10% +0.00% 0 → 0 faster
Tokenize/scratch/complex -45.84% <0.001 -0.09% +0.00% 0 → 0 faster
Tokenize/scratch/insert -42.13% <0.001 +0.02% +0.00% 0 → 0 faster
Normalize/simple_select -4.62% <0.001 -0.20% +0.00% 7 → 7 faster
Normalize/join -12.32% <0.001 -0.60% +0.00% 23 → 23 faster
Normalize/complex -13.74% <0.001 +0.07% +0.00% 69 → 69 faster
TransformParse/simple_select -5.54% <0.001 -0.18% +0.00% 5 → 5 faster
TransformParse/join -15.46% <0.001 -0.42% +0.00% 17 → 17 faster
TransformParse/complex -18.92% <0.001 -0.20% +0.00% 50 → 50 faster
geomean -16.12% <0.001 -0.11%

Breaking Changes

None. Only unexported functions change signature. Token output, spans, errors and allocations are identical.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2

…ue [W2-10]

nextToken and the token readers it dispatches to now write the token
into its slot of the output slice through an out pointer, instead of
returning a 40 B models.Token that was spilled, copied into a 72 B
TokenWithSpan and appended. On amd64 each of those copies reloaded a
just-spilled struct with 16-byte loads that cannot be store-forwarded
(C0: ~39% of amd64 lexer time on those lines).

The slow reference readers and the prefixed/Unicode-escape readers keep
their by-value signatures and are bridged with a field-by-field
storeToken. A failed slot is zeroed so tokBuf[len:cap] stays zero.
Token output, spans and errors are unchanged (goldens pass).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
@vercel

vercel Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
gosqlx-website Ready Ready Preview Nov 2, 2026 9:50pm UTC

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 23s —— View job


Review of PR #926: build tokens in place

  • Read the diff (token_out.go, the tokenize loop, the reader signature changes)
  • Review for correctness, performance, safety and tests

I reviewed this by reading the diff. I did not run the test suite, so I haven't checked the "task check passes" claim myself.

Summary

The change is mechanical and carefully scoped. Readers on the hot path now write into *out and return only error. The slow reference readers keep their by-value signatures and are bridged with storeToken, so the fast-vs-slow differential tests still compare the same things. The measured gains are consistent: -14% on scratch tokenize and -7% on E2E, with allocations unchanged.

What looks correct

  • Slot lifecycle in tokenize: slices.Grow then tokens[:n+1] gives a slot that is zero under the tokBuf[len:cap]-is-zero invariant. On error, *tw = models.TokenWithSpan{} followed by tokens = tokens[:n] restores that invariant. This is the riskiest part of the change, and the error-parity test targets it.
  • setToken on success only: a reader that errors after partial work never writes a half-built token. The comment in token_out.go ("left as it was or partly written") is more conservative than the code actually needs, which is fine.
  • storeToken skips the write on error and relies on the slot being zero. That is safe because every caller passes a freshly zeroed slot. The var inner models.Token in prefixed_string.go is also zero.
  • TestStoreToken_CopiesEveryField uses reflection to catch a new models.Token field being dropped by the bridge. That is a good guard for the one place this change could silently lose data.

Suggestions (non-blocking)

  1. storeToken is a latent trap. It copies the five known fields and does not clear the rest. If a caller ever passes a non-zero out (for example a future reuse of a slot), stale fields would survive. Either add a doc line saying out must be zero, or assert it in a test.
  2. slices.Grow(tokens, 1) vs append growth. Your description says the growth sizes match. Grow rounds up to the size class, which append also does, so capacity should match. A one-line test asserting cap parity with the old path would keep the memory/allocs claim from silently regressing. TestTokenize_GrowsPastEstimate covers the values but not capacity.
  3. The slog debug line now formats tw.Token with %T, which is the same as before and always prints models.Token. It was already a low-value attribute and is not a regression, but it could log tw.Type instead if you touch it again.
  4. Perf evidence: the numbers are from M4 only, and the PR's stated motivation is an amd64 store-forwarding stall. Hold the merge on the amd64 CI perf gate, as you noted, rather than on the M4 results.

Security

No new concerns. There is no new unsafe code, bounds handling is unchanged, and the pointer into the slice is not retained across a Grow (tw is taken after growing).

Verdict

Looks good to merge once the amd64 gate confirms the improvement. The suggestions above are optional hardening.

@github-actions github-actions Bot added documentation Improvements or additions to documentation tokenizer Tokenizer / lexer changes labels Nov 2, 2026
@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Perf gate: REGRESSION (shadow mode: report only, this check never fails)

Base 533148d38172, head afb12ea1cb4f. 10 interleaved rounds, benchtime 200ms, -cpu 1, pinned to CPU 3 (SMT siblings 2-3) with taskset.

  • 1 benchmark(s) slower by 3% or more at p < 0.01
  • 28 benchmark(s) faster by 3% or more

Rules: time regresses at >= +3% with p < 0.01 (Mann-Whitney U), or a suite geomean >= +1.5% (per-round geomeans, same test); allocs/op must not increase; B/op must not rise more than 2%; instructions/op must not rise more than 1.5%. An A/A control with more than 10% of benchmarks moving over 3% is noisy: re-run once, then inconclusive (neutral).

Gated suite (GOGC=off, GOMAXPROCS=1)

benchmark time p A/A B/op allocs/op status
ParseE2E/simple_select -6.65% 0.001 -0.01% +0.00% 5 → 5 faster
ParseE2E/select_where -9.34% <0.001 +0.74% +0.00% 18 → 18 faster
ParseE2E/join -16.68% <0.001 +0.62% +0.00% 17 → 17 faster
ParseE2E/aggregate -13.05% <0.001 +0.53% +0.00% 18 → 18 faster
ParseE2E/subquery -13.27% <0.001 +0.57% +0.00% 14 → 14 faster
ParseE2E/cte -12.56% 0.003 +0.61% +0.00% 21 → 21 faster
ParseE2E/window -14.91% <0.001 -0.54% +0.00% 11 → 11 faster
ParseE2E/complex -18.54% <0.001 -0.34% +0.00% 50 → 50 faster
ParseE2E/insert -17.49% <0.001 +1.35% +0.00% 17 → 17 faster
ParseE2E/update -12.36% 0.002 +1.21% +0.00% 13 → 13 faster
ParseE2E/delete -8.43% 0.011 -0.78% +0.00% 5 → 5 ok
ParseRW/postgresql/medium -9.04% 0.001 +0.63% -0.00% 646 → 646 faster
ParseRW/mysql/medium -12.81% <0.001 +0.64% -0.00% 365 → 365 faster
ParseRW/mariadb/medium -13.62% <0.001 +0.24% -0.00% 523 → 523 faster
ParseRW/sqlserver/medium -12.57% <0.001 +0.91% -0.00% 928 → 928 faster
ParseRW/oracle/medium -12.44% 0.002 +0.07% -0.00% 1093 → 1093 faster
ParseRW/sqlite/medium -13.09% <0.001 -0.33% +0.00% 346 → 346 faster
ParseRW/snowflake/medium -14.47% <0.001 +0.55% +0.00% 777 → 777 faster
ParseRW/clickhouse/medium -13.46% <0.001 +1.05% +0.00% 384 → 384 faster
ParseFresh/simple_select +1.88% 0.436 -0.12% +0.00% 6 → 6 ok
ParseFresh/join -6.08% <0.001 -0.39% +0.00% 18 → 18 faster
ParseFresh/complex -10.42% <0.001 -0.95% +0.00% 51 → 51 faster
PoolGetPut/serial +3.05% 0.005 +1.59% +0.00% 0 → 0 slower
Tokenize/scratch/join -41.68% <0.001 +0.99% +0.00% 0 → 0 faster
Tokenize/scratch/complex -40.93% <0.001 +0.94% +0.00% 0 → 0 faster
Tokenize/scratch/insert -42.53% <0.001 -0.69% +0.00% 0 → 0 faster
Normalize/simple_select -3.41% 0.054 +0.14% +0.00% 7 → 7 ok
Normalize/join -11.24% <0.001 +1.78% +0.00% 23 → 23 faster
Normalize/complex -13.41% <0.001 +1.09% +0.00% 69 → 69 faster
TransformParse/simple_select -7.65% <0.001 +1.32% +0.00% 5 → 5 faster
TransformParse/join -13.21% <0.001 +0.43% +0.00% 17 → 17 faster
TransformParse/complex -16.43% <0.001 +1.60% +0.00% 50 → 50 faster
geomean -14.66% <0.001 +0.48%

A/A control: 0 of 32 benchmarks moved more than 3%.

Instructions per operation (cmd/perfdriver)

case instructions/op status
e2e/aggregate -4.57% ok
e2e/complex -5.62% ok
e2e/cte -4.19% ok
e2e/delete -3.66% ok
e2e/insert -5.07% ok
e2e/join -5.59% ok
e2e/select_where -4.21% ok
e2e/simple_select -3.43% ok
e2e/subquery -4.15% ok
e2e/update -4.88% ok
e2e/window -4.67% ok
rw/postgresql/medium -4.63% ok
Secondary metric: default GC, GOMAXPROCS=1 (informational)
benchmark time p
ParseE2E/simple_select -6.28% <0.001
ParseE2E/select_where -6.63% 0.002
ParseE2E/join -11.72% <0.001
ParseE2E/aggregate -9.29% <0.001
ParseE2E/subquery -7.97% 0.026
ParseE2E/cte -8.10% <0.001
ParseE2E/window -8.46% 0.002
ParseE2E/complex -13.13% <0.001
ParseE2E/insert -12.42% 0.002
ParseE2E/update -7.93% <0.001
ParseE2E/delete -6.24% <0.001
ParseRW/postgresql/medium -8.99% <0.001
ParseRW/mysql/medium -8.81% <0.001
ParseRW/mariadb/medium -9.26% <0.001
ParseRW/sqlserver/medium -9.10% <0.001
ParseRW/oracle/medium -6.34% <0.001
ParseRW/sqlite/medium -8.11% <0.001
ParseRW/snowflake/medium -8.10% <0.001
ParseRW/clickhouse/medium -9.03% <0.001
ParseFresh/simple_select -0.34% 0.326
ParseFresh/join -10.00% <0.001
ParseFresh/complex -13.88% <0.001
PoolGetPut/serial -0.94% 0.739
Tokenize/scratch/join -42.45% <0.001
Tokenize/scratch/complex -42.55% <0.001
Tokenize/scratch/insert -42.96% <0.001
Normalize/simple_select -4.60% 0.009
Normalize/join -10.31% 0.002
Normalize/complex -9.88% 0.002
TransformParse/simple_select -7.52% 0.002
TransformParse/join -12.12% <0.001
TransformParse/complex -12.04% 0.002
geomean -12.50%
Environment
date:        2026-11-02T21:49:55Z
base:        533148d3817286e3ec4c827be5378ebf3016c483
head:        working tree of afb12ea1cb4f00db678dc6a7a78dc5d9f176d691
go:          go version go1.26.1 linux/amd64
kernel:      Linux 7.0.0-1017-azure x86_64
pinning:     pinned to CPU 3 (SMT siblings 2-3) with taskset
clocksource: tsc
load:        1.55, 0.39, 0.13
instr count: cachegrind
perf probe:  Error: No supported events found. The instructions:u event is not supported. 

Architecture:                            x86_64
CPU(s):                                  4
Model name:                              AMD EPYC 9V45 96-Core Processor
Thread(s) per core:                      2
Core(s) per socket:                      2
Socket(s):                               1
Hypervisor vendor:                       Microsoft
L1d cache:                               96 KiB (2 instances)
L1i cache:                               64 KiB (2 instances)
L2 cache:                                2 MiB (2 instances)
L3 cache:                                32 MiB (1 instance)
benchstat (primary metric)
goos: linux
goarch: amd64
pkg: github.com/ajitpratap0/GoSQLX/pkg/fingerprint
cpu: AMD EPYC 9V45 96-Core Processor                
                        │     base     │                head                 │
                        │    sec/op    │   sec/op     vs base                │
Normalize/simple_select    1.393µ ± 3%   1.345µ ± 4%        ~ (p=0.050 n=10)
Normalize/join             4.374µ ± 2%   3.883µ ± 2%  -11.24% (p=0.000 n=10)
Normalize/complex         11.500µ ± 2%   9.958µ ± 3%  -13.41% (p=0.000 n=10)
geomean                    4.122µ        3.732µ        -9.45%

                        │     base     │                 head                  │
                        │     B/op     │     B/op      vs base                 │
Normalize/simple_select     896.0 ± 0%     896.0 ± 0%       ~ (p=1.000 n=10) ¹
Normalize/join            2.680Ki ± 0%   2.680Ki ± 0%       ~ (p=1.000 n=10) ¹
Normalize/complex         7.492Ki ± 0%   7.492Ki ± 0%       ~ (p=1.000 n=10) ¹
geomean                   2.600Ki        2.600Ki       +0.00%
¹ all samples are equal

                        │    base    │                head                 │
                        │ allocs/op  │ allocs/op   vs base                 │
Normalize/simple_select   7.000 ± 0%   7.000 ± 0%       ~ (p=1.000 n=10) ¹
Normalize/join            23.00 ± 0%   23.00 ± 0%       ~ (p=1.000 n=10) ¹
Normalize/complex         69.00 ± 0%   69.00 ± 0%       ~ (p=1.000 n=10) ¹
geomean                   22.31        22.31       +0.00%
¹ all samples are equal

pkg: github.com/ajitpratap0/GoSQLX/pkg/gosqlx
                          │     base     │                 head                 │
                          │    sec/op    │    sec/op     vs base                │
ParseE2E/simple_select       999.9n ± 3%   933.4n ±  3%   -6.65% (p=0.001 n=10)
ParseE2E/select_where        1.949µ ± 3%   1.766µ ±  3%   -9.34% (p=0.000 n=10)
ParseE2E/join                3.003µ ± 2%   2.502µ ±  3%  -16.68% (p=0.000 n=10)
ParseE2E/aggregate           2.165µ ± 4%   1.882µ ±  2%  -13.05% (p=0.000 n=10)
ParseE2E/subquery            1.918µ ± 2%   1.663µ ±  2%  -13.27% (p=0.000 n=10)
ParseE2E/cte                 2.615µ ± 2%   2.286µ ±  3%  -12.56% (p=0.001 n=10)
ParseE2E/window              1.677µ ± 3%   1.427µ ±  4%  -14.91% (p=0.000 n=10)
ParseE2E/complex             8.005µ ± 2%   6.521µ ±  3%  -18.54% (p=0.000 n=10)
ParseE2E/insert              1.369µ ± 1%   1.129µ ±  1%  -17.49% (p=0.000 n=10)
ParseE2E/update              1.194µ ± 3%   1.046µ ±  3%  -12.36% (p=0.001 n=10)
ParseE2E/delete              702.6n ± 9%   643.3n ± 17%        ~ (p=0.011 n=10)
ParseRW/postgresql/medium    97.67µ ± 4%   88.84µ ±  3%   -9.04% (p=0.001 n=10)
ParseRW/mysql/medium         52.03µ ± 1%   45.37µ ±  5%  -12.81% (p=0.000 n=10)
ParseRW/mariadb/medium       74.68µ ± 1%   64.51µ ±  3%  -13.62% (p=0.000 n=10)
ParseRW/sqlserver/medium     140.4µ ± 4%   122.8µ ±  5%  -12.57% (p=0.001 n=10)
ParseRW/oracle/medium        172.8µ ± 3%   151.3µ ±  3%  -12.44% (p=0.002 n=10)
ParseRW/sqlite/medium        45.66µ ± 4%   39.69µ ±  2%  -13.09% (p=0.000 n=10)
ParseRW/snowflake/medium    104.78µ ± 2%   89.61µ ±  2%  -14.47% (p=0.000 n=10)
ParseRW/clickhouse/medium    50.12µ ± 2%   43.37µ ±  3%  -13.46% (p=0.000 n=10)
ParseFresh/simple_select     4.127µ ± 3%   4.205µ ±  3%        ~ (p=0.436 n=10)
ParseFresh/join              6.844µ ± 3%   6.428µ ±  1%   -6.08% (p=0.000 n=10)
ParseFresh/complex           12.28µ ± 2%   11.00µ ±  3%  -10.42% (p=0.000 n=10)
PoolGetPut/serial            58.84n ± 2%   60.64n ±  2%   +3.05% (p=0.005 n=10)
geomean                      7.147µ        6.340µ        -11.29%

                          │     base     │                 head                  │
                          │     B/s      │      B/s       vs base                │
ParseE2E/simple_select      24.81Mi ± 3%   26.56Mi ±  3%   +7.09% (p=0.001 n=10)
ParseE2E/select_where       50.42Mi ± 3%   55.61Mi ±  3%  +10.29% (p=0.000 n=10)
ParseE2E/join               40.01Mi ± 2%   48.03Mi ±  3%  +20.03% (p=0.000 n=10)
ParseE2E/aggregate          37.01Mi ± 4%   42.57Mi ±  2%  +15.01% (p=0.000 n=10)
ParseE2E/subquery           40.79Mi ± 2%   47.02Mi ±  2%  +15.28% (p=0.000 n=10)
ParseE2E/cte                45.22Mi ± 2%   51.72Mi ±  3%  +14.36% (p=0.002 n=10)
ParseE2E/window             53.46Mi ± 3%   62.83Mi ±  4%  +17.53% (p=0.000 n=10)
ParseE2E/complex            45.04Mi ± 2%   55.28Mi ±  3%  +22.75% (p=0.000 n=10)
ParseE2E/insert             52.96Mi ± 1%   64.17Mi ±  1%  +21.16% (p=0.000 n=10)
ParseE2E/update             43.94Mi ± 3%   50.16Mi ±  3%  +14.17% (p=0.001 n=10)
ParseE2E/delete             70.58Mi ± 8%   77.09Mi ± 14%        ~ (p=0.011 n=10)
ParseRW/postgresql/medium   36.74Mi ± 4%   40.38Mi ±  3%   +9.92% (p=0.001 n=10)
ParseRW/mysql/medium        52.02Mi ± 1%   59.66Mi ±  5%  +14.69% (p=0.000 n=10)
ParseRW/mariadb/medium      47.48Mi ± 1%   54.97Mi ±  3%  +15.76% (p=0.000 n=10)
ParseRW/sqlserver/medium    47.38Mi ± 4%   54.19Mi ±  5%  +14.38% (p=0.001 n=10)
ParseRW/oracle/medium       48.52Mi ± 3%   55.41Mi ±  3%  +14.20% (p=0.001 n=10)
ParseRW/sqlite/medium       45.24Mi ± 4%   52.05Mi ±  2%  +15.05% (p=0.000 n=10)
ParseRW/snowflake/medium    48.52Mi ± 2%   56.73Mi ±  2%  +16.92% (p=0.000 n=10)
ParseRW/clickhouse/medium   44.97Mi ± 2%   51.96Mi ±  3%  +15.56% (p=0.000 n=10)
geomean                     45.18Mi        51.89Mi        +14.85%

                          │      base      │                 head                  │
                          │      B/op      │     B/op      vs base                 │
ParseE2E/simple_select        816.0 ± 0%       816.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/select_where       1.523Ki ± 0%     1.523Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/join               2.305Ki ± 0%     2.305Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/aggregate          1.812Ki ± 0%     1.812Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/subquery           1.828Ki ± 0%     1.828Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/cte                2.336Ki ± 0%     2.336Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/window             1.281Ki ± 0%     1.281Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/complex            6.398Ki ± 0%     6.398Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/insert               824.0 ± 0%       824.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/update               736.0 ± 0%       736.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/delete               400.0 ± 0%       400.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/postgresql/medium   70.50Ki ± 0%     70.50Ki ± 0%  -0.00% (p=0.001 n=10)
ParseRW/mysql/medium        43.72Ki ± 0%     43.72Ki ± 0%  -0.00% (p=0.000 n=10)
ParseRW/mariadb/medium      64.82Ki ± 0%     64.82Ki ± 0%  -0.00% (p=0.000 n=10)
ParseRW/sqlserver/medium    90.48Ki ± 0%     90.48Ki ± 0%  -0.00% (p=0.000 n=10)
ParseRW/oracle/medium       113.8Ki ± 0%     113.8Ki ± 0%  -0.00% (p=0.000 n=10)
ParseRW/sqlite/medium       39.89Ki ± 0%     39.89Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/snowflake/medium    83.38Ki ± 0%     83.38Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/clickhouse/medium   37.13Ki ± 0%     37.13Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/simple_select      896.0 ± 0%       896.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/join             2.383Ki ± 0%     2.383Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/complex          6.477Ki ± 0%     6.477Ki ± 0%       ~ (p=1.000 n=10) ¹
PoolGetPut/serial             0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                                  ²                 -0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                          │     base      │                 head                 │
                          │   allocs/op   │  allocs/op   vs base                 │
ParseE2E/simple_select       5.000 ± 0%      5.000 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/select_where        18.00 ± 0%      18.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/join                17.00 ± 0%      17.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/aggregate           18.00 ± 0%      18.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/subquery            14.00 ± 0%      14.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/cte                 21.00 ± 0%      21.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/window              11.00 ± 0%      11.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/complex             50.00 ± 0%      50.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/insert              17.00 ± 0%      17.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/update              13.00 ± 0%      13.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/delete              5.000 ± 0%      5.000 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/postgresql/medium    646.0 ± 0%      646.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/mysql/medium         365.0 ± 0%      365.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/mariadb/medium       523.0 ± 0%      523.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/sqlserver/medium     928.0 ± 0%      928.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/oracle/medium       1.093k ± 0%     1.093k ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/sqlite/medium        346.0 ± 0%      346.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/snowflake/medium     777.0 ± 0%      777.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/clickhouse/medium    384.0 ± 0%      384.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/simple_select     6.000 ± 0%      6.000 ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/join              18.00 ± 0%      18.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/complex           51.00 ± 0%      51.00 ± 0%       ~ (p=1.000 n=10) ¹
PoolGetPut/serial            0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                                 ²                +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                          │    base     │                head                 │
                          │  sec/stmt   │  sec/stmt    vs base                │
ParseRW/postgresql/medium   8.139µ ± 4%   7.404µ ± 3%   -9.03% (p=0.001 n=10)
ParseRW/mysql/medium        6.505µ ± 1%   5.671µ ± 5%  -12.81% (p=0.000 n=10)
ParseRW/mariadb/medium      8.297µ ± 1%   7.168µ ± 3%  -13.62% (p=0.000 n=10)
ParseRW/sqlserver/medium    6.688µ ± 4%   5.847µ ± 5%  -12.57% (p=0.001 n=10)
ParseRW/oracle/medium       7.200µ ± 3%   6.305µ ± 3%  -12.44% (p=0.001 n=10)
ParseRW/sqlite/medium       7.610µ ± 4%   6.614µ ± 2%  -13.09% (p=0.000 n=10)
ParseRW/snowflake/medium    8.060µ ± 2%   6.894µ ± 2%  -14.47% (p=0.000 n=10)
ParseRW/clickhouse/medium   7.160µ ± 2%   6.196µ ± 3%  -13.46% (p=0.000 n=10)
geomean                     7.430µ        6.487µ       -12.70%

pkg: github.com/ajitpratap0/GoSQLX/pkg/sql/tokenizer
                         │     base     │                head                 │
                         │    sec/op    │   sec/op     vs base                │
Tokenize/scratch/join      1360.0n ± 2%   793.1n ± 4%  -41.68% (p=0.000 n=10)
Tokenize/scratch/complex    3.623µ ± 2%   2.140µ ± 2%  -40.93% (p=0.000 n=10)
Tokenize/scratch/insert     660.1n ± 3%   379.4n ± 7%  -42.53% (p=0.000 n=10)
geomean                     1.482µ        863.5n       -41.72%

                         │     base     │                 head                  │
                         │     B/s      │      B/s       vs base                │
Tokenize/scratch/join      88.37Mi ± 2%   151.52Mi ± 4%  +71.47% (p=0.000 n=10)
Tokenize/scratch/complex   99.50Mi ± 2%   168.44Mi ± 2%  +69.29% (p=0.000 n=10)
Tokenize/scratch/insert    109.8Mi ± 3%    191.1Mi ± 6%  +74.01% (p=0.000 n=10)
geomean                    98.83Mi         169.6Mi       +71.58%

                         │     base     │                head                 │
                         │     B/op     │    B/op     vs base                 │
Tokenize/scratch/join      0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
Tokenize/scratch/complex   0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
Tokenize/scratch/insert    0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                               ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                         │     base     │                head                 │
                         │  allocs/op   │ allocs/op   vs base                 │
Tokenize/scratch/join      0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
Tokenize/scratch/complex   0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
Tokenize/scratch/insert    0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                               ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

pkg: github.com/ajitpratap0/GoSQLX/pkg/transform
                             │    base     │                head                 │
                             │   sec/op    │   sec/op     vs base                │
TransformParse/simple_select   988.9n ± 3%   913.2n ± 2%   -7.65% (p=0.000 n=10)
TransformParse/join            3.162µ ± 3%   2.744µ ± 1%  -13.21% (p=0.000 n=10)
TransformParse/complex         8.575µ ± 3%   7.167µ ± 3%  -16.43% (p=0.000 n=10)
geomean                        2.993µ        2.619µ       -12.50%

                             │     base     │                 head                  │
                             │     B/op     │     B/op      vs base                 │
TransformParse/simple_select     816.0 ± 0%     816.0 ± 0%       ~ (p=1.000 n=10) ¹
TransformParse/join            2.305Ki ± 0%   2.305Ki ± 0%       ~ (p=1.000 n=10) ¹
TransformParse/complex         6.398Ki ± 0%   6.398Ki ± 0%       ~ (p=1.000 n=10) ¹
geomean                        2.273Ki        2.273Ki       +0.00%
¹ all samples are equal

                             │    base    │                head                 │
                             │ allocs/op  │ allocs/op   vs base                 │
TransformParse/simple_select   5.000 ± 0%   5.000 ± 0%       ~ (p=1.000 n=10) ¹
TransformParse/join            17.00 ± 0%   17.00 ± 0%       ~ (p=1.000 n=10) ¹
TransformParse/complex         50.00 ± 0%   50.00 ± 0%       ~ (p=1.000 n=10) ¹
geomean                        16.20        16.20       +0.00%
¹ all samples are equal

Run, raw data and benchstat

Resolve the W2-03 conflict in readPunctuation: the zero-copy @PARAM and
@@var fast path (scanAtName) now writes through setToken(out, ...), and
the readIdentifier fallback keeps the out-parameter form. CHANGELOG keeps
both entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 17s —— View job


Review of PR #926: build tokens in place

I reviewed the diff by reading it (token_out.go, the tokenize loop, and the reader signature changes). I did not run the tests myself.

Correctness

  • The slot lifecycle in tokenize is sound. The code grows the slice when len == cap, then reslices to n+1, and takes tw after the grow, so the pointer is never stale. On error it zeroes the slot and truncates to n, which keeps the tokBuf[len:cap]-is-zero invariant.
  • setToken writes only on success, and storeToken writes nothing on error. Together they leave the slot zero when a reader fails.
  • storeToken copies five fields. TestStoreToken_CopiesEveryField uses reflection to catch a new models.Token field being dropped, which is the right guard here.

Suggestions (non-blocking)

  1. storeToken assumes out is already zero, because it does not clear the fields it doesn't copy. The comment on readers write *out only on success ... left as it was or partly written (token_out.go:30) is looser than what the code needs. I'd add a line to the storeToken doc saying out must be zero, so a future caller doesn't reuse a dirty slot.
  2. slices.Grow(tokens, 1) should grow like append, but no test pins the resulting cap. TestTokenize_GrowsPastEstimate checks values and spans, not capacity. A small test asserting cap parity would stop the memory claim from regressing silently. The unchanged B/op and allocs/op in the benchmarks already cover most of this.
  3. The debug log still uses fmt.Sprintf("%T", tw.Token), which always prints models.Token. This is not a regression, but logging tw.Type would be more useful if you touch the line again.

Performance
The amd64 gate passes: geomean −15.9%, scratch tokenize −42% to −46%, allocs/op unchanged, and the A/A control is clean. That confirms the store-forwarding explanation. Because the gate runs in shadow mode, it reports only and never blocks the merge.

Security
I found no new concerns. There is no unsafe code, and bounds handling is unchanged.

Verdict
Looks good to merge. The suggestions above are optional hardening.

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 19s —— View job


Review of PR #926: build tokens in place

I read token_out.go and the new tokenize loop. I did not run the tests. The slot handling is correct and I have no blocking concerns.

Correctness

  • slices.Grow runs before tw := &tokens[n], so the pointer is never stale. On error the slot is zeroed and tokens is truncated to n, which keeps the invariant that tokBuf[len:cap] is zero. TestTokenize_ErrorLeavesSpareSlotsZero covers this across the reader families.
  • storeToken copies all five models.Token fields. TestStoreToken_CopiesEveryField uses reflection, so a field added later can't be silently dropped.

Suggestions (non-blocking)

  1. Contradictory comment (token_out.go:30-31). It says readers "write *out only on success", then says that on error *out is "left as it was or partly written". Pick one. setToken and storeToken do write only on success, so I'd drop the "partly written" half. Keeping it makes a reader of the code think a failed slot can be dirty, and the caller zeroes it either way.
  2. storeToken needs a zero out (token_out.go:44). It writes five fields and doesn't clear anything else. That is safe today because every caller passes a freshly zeroed slot. A doc line saying out must be zero would keep a future caller from reusing a dirty slot.
  3. Capacity not pinned. The claim that slices.Grow(tokens, 1) grows like append is only checked indirectly, through unchanged B/op and allocs/op in the benchmarks. A small cap parity assertion would pin it.
  4. Debug log (tokenizer.go:734). fmt.Sprintf("%T", tw.Token) always prints models.Token. It was already like this, so this is not a regression. It is only worth fixing, by logging tw.Type, if you touch the line anyway.

Performance
The amd64 perf gate passes: suite geomean −16.1%, Tokenize/scratch −42% to −46%, and allocs/op and B/op unchanged. The A/A control is clean (0 of 32 benchmarks moved more than 3%), and instructions/op are down 3–5%. The result is consistent with the store-forwarding explanation in the description.

Security
No new concerns. There is no unsafe code, and the bounds handling is unchanged.

Verdict: Good to merge. The suggestions are optional, and #1 is the one I'd fix.
· branch perf/w2-10-tokens-in-place

@ajitpratap0
ajitpratap0 merged commit fd7eb53 into main Nov 2, 2026
48 checks passed
ajitpratap0 added a commit that referenced this pull request Nov 2, 2026
…all-slices

Brings in #926 (tokens built in place) and #930 (O(1) ReleaseAST).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2

This branch was successfully deployed

1 active deployment
Preview — 319865ce Deployed Nov 2, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation tokenizer Tokenizer / lexer changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant