Skip to content

perf(tokenizer): single fused-hash keyword table [W2-01] - #920

Merged
ajitpratap0 merged 5 commits into
mainfrom
perf/w2-01-keyword-table
Nov 1, 2026
Merged

perf(tokenizer): single fused-hash keyword table [W2-01]#920
ajitpratap0 merged 5 commits into
mainfrom
perf/w2-01-keyword-table

Conversation

@ajitpratap0

@ajitpratap0 ajitpratap0 commented Nov 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Summary

PLAN W2-01 (TOK-1). Replaces the two Go map lookups per identifier (keywordTokenTypes, compoundKeywordStarts) with one 512-slot open-addressed table generated at init from both maps. FNV-1a hashing is fused into the ASCII upper-casing loop, so a word costs one hash pass plus typically one probe. Non-ASCII words keep the existing strings.ToUpper + map path. IsKeyword also uses the table. Ported from verify-tokenizer/TOK-1.patch into the existing keyword_lookup.go.

The parser-remap slot reserved by the plan is NOT a struct field yet: golangci-lint (unused) rejects an unread field. W3-03 adds it to kwEntry (noted in a comment).

No exported API change; token output unchanged (token goldens pass untouched, no golden diffs).

Test Plan

  • New TestKwTableMatchesMaps: every key in both maps resolves identically (type, isKeyword, compoundStart) in upper/lower/mixed case; non-keywords miss; no duplicate keys; load factor < 0.5.
  • New TestIsKeywordMatchesMap: table-backed IsKeyword vs the map incl. non-ASCII and over-long words.
  • task check: pass. task test:cbinding: pass. go test -count=1 ./pkg/compatibility/...: pass. staticcheck ./pkg/sql/tokenizer/: clean.
  • Local task bench:ab BASE=origin/main re-run on the final head (M4, shared machine, -r 10 -t 200ms, INDICATIVE ONLY; CI is authoritative): Tokenize/scratch sec/op geomean -16.1% (all 11 queries p=0.000), ParseE2E sec/op geomean -8.2% (all 11 p<=0.001); allocs/op and B/op unchanged. Local gate verdict geomean -12.24%. Full benchstat in the details block below.
  • Probe layout is now deterministic (hot keywords inserted first, rest sorted); TestKeywordTableHotKeysAtHome asserts the hot keys sit in their home slot. Tokenizer.keywordAt now goes through IsKeyword (single source of truth).
  • CI perf-gate (shadow, amd64, authoritative) at the previous head 881c167, PASS: suite geomean -6.72% (the only geomean the gate reports; p<0.001, A/A +0.01%, 0 of 26 moved >3%). ParseE2E -4.0% to -7.5% on all 11 queries (geomean -5.3%, computed from the gate's per-query rows; plan threshold -4%: met). Tokenize/scratch join -21.7%, complex -20.4%, insert -13.2% (geomean -18.5% computed from the 3 gated rows; plan threshold <= -11%: met). ParseRW/*/medium -5.5% to -6.7%. allocs/op and B/op unchanged, instructions/op -4.7% to -8.8%, no regression. The gate re-runs on the latest head (deterministic layout); see the sticky perf-gate comment on this PR for the current table.
Local benchstat (sec/op), base origin/main vs head 
                       │   sec/op    │   sec/op     vs base                │
ParseE2E/simple_select   717.2n ± 1%   668.2n ± 1%   -6.83% (p=0.000 n=10)
ParseE2E/select_where    1.751µ ± 1%   1.599µ ± 2%   -8.65% (p=0.000 n=10)
ParseE2E/join            2.780µ ± 1%   2.494µ ± 4%  -10.29% (p=0.000 n=10)
ParseE2E/aggregate       1.909µ ± 3%   1.781µ ± 5%   -6.68% (p=0.000 n=10)
ParseE2E/subquery        1.653µ ± 1%   1.510µ ± 2%   -8.62% (p=0.000 n=10)
ParseE2E/cte             2.347µ ± 2%   2.127µ ± 1%   -9.39% (p=0.000 n=10)
ParseE2E/window          1.543µ ± 3%   1.415µ ± 2%   -8.30% (p=0.000 n=10)
ParseE2E/complex         7.440µ ± 7%   6.621µ ± 3%  -11.01% (p=0.000 n=10)
ParseE2E/insert          1.255µ ± 2%   1.180µ ± 2%   -5.98% (p=0.000 n=10)
ParseE2E/update          1.094µ ± 1%   1.016µ ± 2%   -7.13% (p=0.001 n=10)
ParseE2E/delete          708.9n ± 1%   656.8n ± 4%   -7.36% (p=0.000 n=10)
geomean                  1.674µ        1.536µ        -8.22%

                               │    sec/op    │   sec/op     vs base                │
Tokenize/scratch/simple_select    301.9n ± 1%   252.5n ± 1%  -16.36% (p=0.000 n=10)
Tokenize/scratch/select_where     793.8n ± 2%   669.0n ± 1%  -15.72% (p=0.000 n=10)
Tokenize/scratch/join             1.346µ ± 1%   1.071µ ± 1%  -20.43% (p=0.000 n=10)
Tokenize/scratch/aggregate        785.8n ± 2%   671.5n ± 1%  -14.54% (p=0.000 n=10)
Tokenize/scratch/subquery         686.4n ± 1%   560.2n ± 1%  -18.39% (p=0.000 n=10)
Tokenize/scratch/cte             1014.5n ± 1%   831.7n ± 1%  -18.02% (p=0.000 n=10)
Tokenize/scratch/window           755.4n ± 3%   633.0n ± 6%  -16.20% (p=0.000 n=10)
Tokenize/scratch/complex          3.518µ ± 3%   2.848µ ± 2%  -19.04% (p=0.000 n=10)
Tokenize/scratch/insert           609.1n ± 1%   543.9n ± 2%  -10.70% (p=0.000 n=10)
Tokenize/scratch/update           518.6n ± 1%   448.6n ± 1%  -13.51% (p=0.000 n=10)
Tokenize/scratch/delete           365.6n ± 1%   315.9n ± 2%  -13.59% (p=0.000 n=10)
geomean                           767.2n        643.7n       -16.09%

Breaking Changes

None.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2

@vercel

vercel Bot commented Nov 1, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
gosqlx-website Ready Ready Preview Nov 1, 2026 5:07pm UTC

@github-actions

github-actions Bot commented Nov 1, 2026 •
edited
Loading

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 28s —— View job

Review of PR #920: fused-hash keyword table

Verdict: looks correct and low-risk. I have a few minor suggestions and no blockers. I read the diff and grepped for related uses. I did not run the tests or benchmarks.

  • Read the diff
  • Check correctness, concurrency, tests and lint concerns

Correctness

  • Table build. init() inserts every key from both maps. A key in both maps gets one slot with isKeyword and compoundStart merged. Linear probing is correct. The table has 512 slots and the tests assert a load factor under 0.5, so kwLookup always reaches an unused slot and terminates.
  • Init order. init() in keyword_lookup.go reads keywordTokenTypes and compoundKeywordStarts. Go finishes package-level var initialization before any init() runs, so this is safe, provided both maps are plain var initializers and not filled in another init(). If another file fills them in its own init(), the order depends on filename. Please confirm. The TestKwTableMatchesMaps equivalence check would catch a problem here.
  • Case handling. asciiUpperHash hashes the upper-cased bytes, so it matches how keys are inserted. Non-ASCII words return ok=false and fall back to the old strings.ToUpper path. The non-ASCII test cases cover the Kelvin-sign style folds (ſelect, ınsert).
  • Behaviour preserved. In readIdentifier, tokenType is only read when isKeyword is true. A compound-start-only entry leaves typ as zero, which matches the old map-miss result.
  • Concurrency. The table is written only during init and is read-only afterwards, so there is no race.
  • Allocations. e.key == string(key) is compared without allocating (the compiler optimizes this form). The existing zero-alloc test in TestIsKeyword covers IsKeyword.

Suggestions (non-blocking)

  1. Hash/lookup duplication. kwHash is used only for insertion, and asciiUpperHash re-implements the same FNV loop. They could drift apart. TestKwTableMatchesMaps would catch that, since every key is looked up through the fast path, so this is covered. A short comment in kwHash saying it must match asciiUpperHash would help.
  2. Table size guard. The load-factor test is a good safety net. If someone adds enough keywords to fill the table, they get a test failure and not an infinite loop. init() itself would still hang if the table became full. A cheap panic when the key count exceeds kwTableSize/2 would make that failure explicit.
  3. Stale helper. asciiUpper is still used at tokenizer.go:1163 and tokenizer.go:2034, so there is no dead-code lint risk. The code at line 1163 (compound-keyword second-word matching) still uses map lookups, so compound handling is only partly on the fast path. That is fine for this PR, but it could be a follow-up.
  4. Cache footprint. kwEntry is about 40 bytes (a 16-byte string, a type, and three bools), so the table is about 20 KB. That is a little large for L1 but fine for L2. Packing the flags into a bitmask or storing keys in a separate array could improve locality. This is a micro-optimization that should wait for real benchmark data.

Benchmarks and performance

The PR notes the local numbers are indicative only (p=0.28 for tokenize and p=0.63 for E2E). The join query regressed by 15.6%, so I'd check that one on the CI perf-gate. Probe chains for identifier-heavy queries may be longer than expected, though FNV-1a at 512 slots should usually probe once. I'd wait for the shadow-gate verdict before treating the speedup as confirmed.

Tests

TestKwTableMatchesMaps and TestIsKeywordMatchesMap give good coverage: case variants, non-ASCII, over-long words, duplicate keys and load factor. Existing token goldens are untouched.

Security

No concerns. There is no external input to the table, and the buffer sizes are bounded by keywordKeyBufLen.

Other

The CHANGELOG entry is present.
· branch perf/w2-01-keyword-table

@github-actions github-actions Bot added documentation Improvements or additions to documentation tokenizer Tokenizer / lexer changes labels Nov 1, 2026
@github-actions

github-actions Bot commented Nov 1, 2026 •
edited
Loading

Copy link
Copy Markdown

Perf gate: PASS (shadow mode: report only, this check never fails)

Base c05c5fd54904, head 38b59c84d52e. 10 interleaved rounds, benchtime 200ms, -cpu 1, pinned to CPU 3 (SMT siblings 2-3) with taskset.

  • 22 benchmark(s) faster by 3% or more

Rules: time regresses at >= +3% with p < 0.01 (Mann-Whitney U), or a suite geomean >= +1.5% (per-round geomeans, same test); allocs/op must not increase; B/op must not rise more than 2%; instructions/op must not rise more than 1.5%. An A/A control with more than 10% of benchmarks moving over 3% is noisy: re-run once, then inconclusive (neutral).

Gated suite (GOGC=off, GOMAXPROCS=1)

benchmark time p A/A B/op allocs/op status
ParseE2E/simple_select -6.61% 0.003 -0.26% +0.00% 5 → 5 faster
ParseE2E/select_where -8.29% <0.001 +0.20% +0.00% 18 → 18 faster
ParseE2E/join -10.80% <0.001 +0.01% +0.00% 17 → 17 faster
ParseE2E/aggregate -4.67% <0.001 +0.13% +0.00% 18 → 18 faster
ParseE2E/subquery -7.48% <0.001 -0.18% +0.00% 14 → 14 faster
ParseE2E/cte -7.28% <0.001 -0.32% +0.00% 21 → 21 faster
ParseE2E/window -7.60% <0.001 -0.01% +0.00% 11 → 11 faster
ParseE2E/complex -9.66% <0.001 +0.12% +0.00% 50 → 50 faster
ParseE2E/insert -8.45% <0.001 +0.12% +0.00% 17 → 17 faster
ParseE2E/update -5.97% <0.001 -0.24% +0.00% 13 → 13 faster
ParseE2E/delete -2.11% <0.001 -0.32% +0.00% 5 → 5 ok
ParseRW/postgresql/medium -5.68% <0.001 -0.52% -0.00% 646 → 646 faster
ParseRW/mysql/medium -6.99% <0.001 +0.03% +0.00% 365 → 365 faster
ParseRW/mariadb/medium -7.15% <0.001 -0.33% -0.00% 523 → 523 faster
ParseRW/sqlserver/medium -4.28% 0.002 +0.00% -0.00% 936 → 936 faster
ParseRW/oracle/medium -5.25% <0.001 -0.20% -0.00% 1093 → 1093 faster
ParseRW/sqlite/medium -7.83% <0.001 +0.07% +0.00% 346 → 346 faster
ParseRW/snowflake/medium -6.38% <0.001 +0.09% +0.00% 777 → 777 faster
ParseRW/clickhouse/medium -7.72% <0.001 +0.17% +0.00% 384 → 384 faster
ParseFresh/simple_select -0.62% 0.004 -0.07% +0.00% 6 → 6 ok
ParseFresh/join -2.14% 0.001 -0.01% +0.00% 18 → 18 ok
ParseFresh/complex -4.04% <0.001 -0.04% +0.00% 51 → 51 faster
PoolGetPut/serial +0.04% 0.570 -0.03% +0.00% 0 → 0 ok
Tokenize/scratch/join -18.64% <0.001 +0.45% +0.00% 0 → 0 faster
Tokenize/scratch/complex -16.96% <0.001 +0.38% +0.00% 0 → 0 faster
Tokenize/scratch/insert -10.26% <0.001 +0.99% +0.00% 0 → 0 faster
geomean -7.13% <0.001 +0.01%

A/A control: 0 of 26 benchmarks moved more than 3%.

Instructions per operation (cmd/perfdriver)

case instructions/op status
e2e/aggregate -5.43% ok
e2e/complex -8.69% ok
e2e/cte -7.44% ok
e2e/delete -6.75% ok
e2e/insert -5.37% ok
e2e/join -9.73% ok
e2e/select_where -7.03% ok
e2e/simple_select -7.66% ok
e2e/subquery -7.74% ok
e2e/update -6.56% ok
e2e/window -8.13% ok
rw/postgresql/medium -7.39% ok
Secondary metric: default GC, GOMAXPROCS=1 (informational)
benchmark time p
ParseE2E/simple_select -6.15% <0.001
ParseE2E/select_where -6.64% <0.001
ParseE2E/join -7.72% <0.001
ParseE2E/aggregate -4.39% <0.001
ParseE2E/subquery -5.29% <0.001
ParseE2E/cte -6.78% <0.001
ParseE2E/window -5.40% <0.001
ParseE2E/complex -6.30% 0.002
ParseE2E/insert -8.49% <0.001
ParseE2E/update -6.48% <0.001
ParseE2E/delete -1.89% <0.001
ParseRW/postgresql/medium -4.96% <0.001
ParseRW/mysql/medium -3.56% <0.001
ParseRW/mariadb/medium -3.96% <0.001
ParseRW/sqlserver/medium -3.03% <0.001
ParseRW/oracle/medium -2.66% <0.001
ParseRW/sqlite/medium -4.50% <0.001
ParseRW/snowflake/medium -4.53% <0.001
ParseRW/clickhouse/medium -5.27% <0.001
ParseFresh/simple_select -0.91% <0.001
ParseFresh/join -2.37% <0.001
ParseFresh/complex -4.01% <0.001
PoolGetPut/serial +0.26% 0.029
Tokenize/scratch/join -18.89% <0.001
Tokenize/scratch/complex -17.15% <0.001
Tokenize/scratch/insert -10.51% <0.001
geomean -5.93%
Environment 
date:        2026-11-01T17:10:04Z
base:        c05c5fd549046b71ab35029a0dac0cbd91f22d27
head:        working tree of 38b59c84d52e43f7e223668a31c3149914c3d52f
go:          go version go1.26.1 linux/amd64
kernel:      Linux 7.0.0-1017-azure x86_64
pinning:     pinned to CPU 3 (SMT siblings 2-3) with taskset
clocksource: tsc
load:        1.77, 0.46, 0.16
instr count: cachegrind
perf probe:  Error: No supported events found. The instructions:u event is not supported. 

Architecture:                            x86_64
CPU(s):                                  4
Model name:                              Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
Thread(s) per core:                      2
Core(s) per socket:                      2
Socket(s):                               1
CPU(s) scaling MHz:                      115%
Hypervisor vendor:                       Microsoft
L1d cache:                               96 KiB (2 instances)
L1i cache:                               64 KiB (2 instances)
L2 cache:                                2.5 MiB (2 instances)
L3 cache:                                48 MiB (1 instance)
benchstat (primary metric) 
goos: linux
goarch: amd64
pkg: github.com/ajitpratap0/GoSQLX/pkg/gosqlx
cpu: Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
                          │     base     │                head                 │
                          │    sec/op    │   sec/op     vs base                │
ParseE2E/simple_select       1.898µ ± 1%   1.772µ ± 1%   -6.61% (p=0.001 n=10)
ParseE2E/select_where        4.076µ ± 0%   3.738µ ± 1%   -8.29% (p=0.000 n=10)
ParseE2E/join                6.014µ ± 1%   5.364µ ± 0%  -10.80% (p=0.000 n=10)
ParseE2E/aggregate           4.495µ ± 1%   4.285µ ± 0%   -4.67% (p=0.000 n=10)
ParseE2E/subquery            3.897µ ± 0%   3.605µ ± 0%   -7.48% (p=0.000 n=10)
ParseE2E/cte                 5.285µ ± 0%   4.900µ ± 1%   -7.28% (p=0.000 n=10)
ParseE2E/window              3.533µ ± 1%   3.264µ ± 0%   -7.60% (p=0.000 n=10)
ParseE2E/complex             15.59µ ± 0%   14.09µ ± 0%   -9.66% (p=0.000 n=10)
ParseE2E/insert              2.823µ ± 2%   2.584µ ± 4%   -8.45% (p=0.000 n=10)
ParseE2E/update              2.481µ ± 1%   2.333µ ± 1%   -5.97% (p=0.000 n=10)
ParseE2E/delete              1.564µ ± 1%   1.530µ ± 0%   -2.11% (p=0.000 n=10)
ParseRW/postgresql/medium    180.0µ ± 0%   169.8µ ± 0%   -5.68% (p=0.000 n=10)
ParseRW/mysql/medium        103.13µ ± 0%   95.93µ ± 1%   -6.99% (p=0.000 n=10)
ParseRW/mariadb/medium       151.2µ ± 1%   140.4µ ± 0%   -7.15% (p=0.000 n=10)
ParseRW/sqlserver/medium     317.8µ ± 1%   304.2µ ± 4%   -4.28% (p=0.002 n=10)
ParseRW/oracle/medium        386.6µ ± 1%   366.3µ ± 1%   -5.25% (p=0.000 n=10)
ParseRW/sqlite/medium        87.17µ ± 1%   80.34µ ± 0%   -7.83% (p=0.000 n=10)
ParseRW/snowflake/medium     222.5µ ± 1%   208.3µ ± 1%   -6.38% (p=0.000 n=10)
ParseRW/clickhouse/medium    99.37µ ± 0%   91.70µ ± 0%   -7.72% (p=0.000 n=10)
ParseFresh/simple_select     10.08µ ± 0%   10.02µ ± 0%   -0.62% (p=0.002 n=10)
ParseFresh/join              22.76µ ± 0%   22.27µ ± 1%   -2.14% (p=0.001 n=10)
ParseFresh/complex           34.33µ ± 0%   32.94µ ± 0%   -4.04% (p=0.000 n=10)
PoolGetPut/serial            87.68n ± 0%   87.71n ± 0%        ~ (p=0.565 n=10)
geomean                      15.04µ        14.14µ        -5.99%

                          │     base     │                 head                 │
                          │     B/s      │     B/s       vs base                │
ParseE2E/simple_select      13.07Mi ± 1%   13.99Mi ± 1%   +7.08% (p=0.001 n=10)
ParseE2E/select_where       24.10Mi ± 0%   26.27Mi ± 1%   +9.02% (p=0.000 n=10)
ParseE2E/join               19.98Mi ± 1%   22.40Mi ± 0%  +12.12% (p=0.000 n=10)
ParseE2E/aggregate          17.82Mi ± 1%   18.70Mi ± 0%   +4.92% (p=0.000 n=10)
ParseE2E/subquery           20.07Mi ± 0%   21.69Mi ± 0%   +8.08% (p=0.000 n=10)
ParseE2E/cte                22.37Mi ± 0%   24.13Mi ± 1%   +7.86% (p=0.000 n=10)
ParseE2E/window             25.38Mi ± 1%   27.47Mi ± 0%   +8.23% (p=0.000 n=10)
ParseE2E/complex            23.12Mi ± 0%   25.59Mi ± 0%  +10.68% (p=0.000 n=10)
ParseE2E/insert             25.68Mi ± 1%   28.05Mi ± 4%   +9.23% (p=0.000 n=10)
ParseE2E/update             21.14Mi ± 1%   22.48Mi ± 1%   +6.34% (p=0.000 n=10)
ParseE2E/delete             31.72Mi ± 1%   32.40Mi ± 0%   +2.15% (p=0.000 n=10)
ParseRW/postgresql/medium   19.93Mi ± 0%   21.13Mi ± 0%   +6.00% (p=0.000 n=10)
ParseRW/mysql/medium        26.25Mi ± 0%   28.21Mi ± 1%   +7.50% (p=0.000 n=10)
ParseRW/mariadb/medium      23.45Mi ± 1%   25.26Mi ± 0%   +7.71% (p=0.000 n=10)
ParseRW/sqlserver/medium    20.94Mi ± 1%   21.87Mi ± 4%   +4.46% (p=0.001 n=10)
ParseRW/oracle/medium       21.68Mi ± 1%   22.88Mi ± 1%   +5.52% (p=0.000 n=10)
ParseRW/sqlite/medium       23.70Mi ± 1%   25.71Mi ± 0%   +8.47% (p=0.000 n=10)
ParseRW/snowflake/medium    22.85Mi ± 1%   24.41Mi ± 1%   +6.82% (p=0.000 n=10)
ParseRW/clickhouse/medium   22.68Mi ± 0%   24.58Mi ± 0%   +8.37% (p=0.000 n=10)
geomean                     22.09Mi        23.72Mi        +7.38%

                          │      base      │                 head                  │
                          │      B/op      │     B/op      vs base                 │
ParseE2E/simple_select        816.0 ± 0%       816.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/select_where       1.523Ki ± 0%     1.523Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/join               2.305Ki ± 0%     2.305Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/aggregate          1.812Ki ± 0%     1.812Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/subquery           1.828Ki ± 0%     1.828Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/cte                2.336Ki ± 0%     2.336Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/window             1.281Ki ± 0%     1.281Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/complex            6.399Ki ± 0%     6.399Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/insert               824.0 ± 0%       824.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/update               736.0 ± 0%       736.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/delete               400.0 ± 0%       400.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/postgresql/medium   70.52Ki ± 0%     70.52Ki ± 0%  -0.00% (p=0.000 n=10)
ParseRW/mysql/medium        43.73Ki ± 0%     43.73Ki ± 0%       ~ (p=0.087 n=10)
ParseRW/mariadb/medium      64.85Ki ± 0%     64.84Ki ± 0%  -0.00% (p=0.000 n=10)
ParseRW/sqlserver/medium    90.60Ki ± 0%     90.60Ki ± 0%  -0.00% (p=0.000 n=10)
ParseRW/oracle/medium       113.8Ki ± 0%     113.8Ki ± 0%  -0.00% (p=0.000 n=10)
ParseRW/sqlite/medium       39.89Ki ± 0%     39.89Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/snowflake/medium    83.38Ki ± 0%     83.38Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/clickhouse/medium   37.13Ki ± 0%     37.13Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/simple_select      896.0 ± 0%       896.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/join             2.383Ki ± 0%     2.383Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/complex          6.477Ki ± 0%     6.477Ki ± 0%       ~ (p=1.000 n=10) ¹
PoolGetPut/serial             0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                                  ²                 -0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                          │     base      │                 head                 │
                          │   allocs/op   │  allocs/op   vs base                 │
ParseE2E/simple_select       5.000 ± 0%      5.000 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/select_where        18.00 ± 0%      18.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/join                17.00 ± 0%      17.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/aggregate           18.00 ± 0%      18.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/subquery            14.00 ± 0%      14.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/cte                 21.00 ± 0%      21.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/window              11.00 ± 0%      11.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/complex             50.00 ± 0%      50.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/insert              17.00 ± 0%      17.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/update              13.00 ± 0%      13.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/delete              5.000 ± 0%      5.000 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/postgresql/medium    646.0 ± 0%      646.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/mysql/medium         365.0 ± 0%      365.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/mariadb/medium       523.0 ± 0%      523.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/sqlserver/medium     936.0 ± 0%      936.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/oracle/medium       1.093k ± 0%     1.093k ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/sqlite/medium        346.0 ± 0%      346.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/snowflake/medium     777.0 ± 0%      777.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/clickhouse/medium    384.0 ± 0%      384.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/simple_select     6.000 ± 0%      6.000 ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/join              18.00 ± 0%      18.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/complex           51.00 ± 0%      51.00 ± 0%       ~ (p=1.000 n=10) ¹
PoolGetPut/serial            0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                                 ²                +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                          │    base     │                head                │
                          │  sec/stmt   │  sec/stmt    vs base               │
ParseRW/postgresql/medium   15.00µ ± 0%   14.15µ ± 0%  -5.68% (p=0.000 n=10)
ParseRW/mysql/medium        12.89µ ± 0%   11.99µ ± 1%  -6.99% (p=0.000 n=10)
ParseRW/mariadb/medium      16.80µ ± 1%   15.60µ ± 0%  -7.15% (p=0.000 n=10)
ParseRW/sqlserver/medium    15.13µ ± 1%   14.49µ ± 4%  -4.28% (p=0.002 n=10)
ParseRW/oracle/medium       16.11µ ± 1%   15.26µ ± 1%  -5.25% (p=0.000 n=10)
ParseRW/sqlite/medium       14.53µ ± 1%   13.39µ ± 0%  -7.83% (p=0.000 n=10)
ParseRW/snowflake/medium    17.11µ ± 1%   16.02µ ± 1%  -6.38% (p=0.000 n=10)
ParseRW/clickhouse/medium   14.20µ ± 0%   13.10µ ± 0%  -7.72% (p=0.000 n=10)
geomean                     15.16µ        14.19µ       -6.42%

pkg: github.com/ajitpratap0/GoSQLX/pkg/sql/tokenizer
                         │    base     │                head                 │
                         │   sec/op    │   sec/op     vs base                │
Tokenize/scratch/join      2.785µ ± 1%   2.266µ ± 0%  -18.64% (p=0.000 n=10)
Tokenize/scratch/complex   7.227µ ± 2%   6.001µ ± 1%  -16.96% (p=0.000 n=10)
Tokenize/scratch/insert    1.268µ ± 1%   1.138µ ± 0%  -10.26% (p=0.000 n=10)
geomean                    2.944µ        2.491µ       -15.36%

                         │     base     │                 head                 │
                         │     B/s      │     B/s       vs base                │
Tokenize/scratch/join      43.15Mi ± 1%   53.04Mi ± 0%  +22.91% (p=0.000 n=10)
Tokenize/scratch/complex   49.88Mi ± 2%   60.07Mi ± 1%  +20.43% (p=0.000 n=10)
Tokenize/scratch/insert    57.20Mi ± 1%   63.71Mi ± 0%  +11.38% (p=0.000 n=10)
geomean                    49.75Mi        58.77Mi       +18.13%

                         │     base     │                head                 │
                         │     B/op     │    B/op     vs base                 │
Tokenize/scratch/join      0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
Tokenize/scratch/complex   0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
Tokenize/scratch/insert    0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                               ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                         │     base     │                head                 │
                         │  allocs/op   │ allocs/op   vs base                 │
Tokenize/scratch/join      0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
Tokenize/scratch/complex   0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
Tokenize/scratch/insert    0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                               ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

Run, raw data and benchstat

@github-actions

github-actions Bot commented Nov 1, 2026 •
edited
Loading

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 22s —— View job

Review of PR #920: fused-hash keyword table

Verdict: correct and low-risk. I found no blockers. I read the diff of keyword_lookup.go and tokenizer.go. I did not run the tests. I rely on the CI perf-gate numbers for performance.

Correctness

  • Table build. init() merges both maps into one key set, so a key in both gets a single slot with isKeyword and compoundStart merged. Only compoundKeywordStarts entries that are true are inserted, and e.compoundStart = compoundKeywordStarts[k] matches that.
  • Termination. init() panics if the key count exceeds kwTableSize/2. That addresses the earlier review's "table full" concern: a full table fails loudly at startup, so kwLookup always reaches an unused slot.
  • Deterministic layout. Hot keys go in first, then the rest in sorted order. Probe placement is the same on every run, which also keeps benchmarks stable.
  • Init order. init() reads the two maps, which must be plain var initializers. If another file filled them in its own init(), the order would depend on filename. TestKwTableMatchesMaps would catch that.
  • Case handling. asciiUpperHash hashes the upper-cased bytes, the same bytes kwHash hashed at insert time. Non-ASCII words fall back to the existing strings.ToUpper map path.
  • readIdentifier. tokenType is only read when isKeyword is true, so a compound-start-only entry (zero typ) behaves like the old map miss.
  • keywordAt. The new IsKeyword(word) && isASCII(word) keeps the old behavior. The old code returned false for any non-ASCII word, and the isASCII guard preserves that even though IsKeyword accepts folds like ſelect. The check is slightly indirect; a one-line comment explaining why isASCII is there would help.
  • Concurrency. The table is written only in init() and read-only afterwards, so there are no races. e.key == string(key) does not allocate.

Suggestions (non-blocking)

  1. Hash duplication. kwHash and asciiUpperHash implement the same FNV-1a loop. The equivalence test would catch drift, but a comment in kwHash saying it must match asciiUpperHash would help. Alternatively, have kwHash call asciiUpperHash on a stack buffer.
  2. Compound second-word lookup. tokenizer.go still uses map lookups through asciiUpper for compound-keyword second words. That is a reasonable follow-up.
  3. Entry size. kwEntry is about 32–40 bytes, so the table is roughly 16–20 KB. This is fine for L2. Packing the flags into one byte would be a micro-optimization to consider only if benchmarks show a need.
  4. join regression in the local bench. The CI perf-gate shows Tokenize/scratch/join at -21.7% with p<0.001. That is authoritative, and the local +15.6% was noise.

Performance

The perf-gate (shadow) passes. The suite geomean is -6.72%, ParseE2E is -4% to -7.5% on every query, and Tokenize/scratch is -13% to -22%. allocs/op and B/op are unchanged, and the A/A control is clean.

Tests

TestKwTableMatchesMaps covers case variants, misses, duplicate keys and load factor. TestIsKeywordMatchesMap covers non-ASCII and over-long words. Together they are good equivalence coverage. If you want extra cover, add a case asserting that keywordAt rejects non-ASCII folds such as ſelect.

Security

No concerns. The table is built from static data, and the stack buffers are bounded by keywordKeyBufLen.

The CHANGELOG entry is present.
· branch perf/w2-01-keyword-table

@github-actions

github-actions Bot commented Nov 1, 2026 •
edited
Loading

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 26s —— View job

Review of PR #920: fused-hash keyword table

Verdict: correct and low-risk, no blockers. I read the diff for keyword_lookup.go and tokenizer.go. I did not run the tests or benchmarks. The CI perf-gate result is strong: suite geomean -6.72% with no allocation change.

An earlier review on this PR suggested a panic guard on table load. The current head already has it (len(keys) > kwTableSize/2), so init() can't loop forever. Keys are also inserted in a fixed order, so the layout is the same in every process.

Correctness

  • Table build. A key in both maps gets one slot, with isKeyword and compoundStart merged. Compound-start-only entries leave typ zero. readIdentifier only reads typ when isKeyword is true, so behaviour matches the old map miss.
  • Init order. init() reads the two maps. This is safe as long as both are plain package-level var initializers. If another file fills them in its own init(), the result depends on file order. TestKwTableMatchesMaps would catch that.
  • Concurrency. The table is written only in init() and is read-only afterwards, so there is no race.
  • Termination. The load factor is under 0.5, so the probe loop always reaches an unused slot.
  • keywordAt (tokenizer.go:2033). The new IsKeyword(word) && isASCII(word) preserves the old behaviour, where a non-ASCII word returned false. IsKeyword could return true through the non-ASCII strings.ToUpper path (for example a Kelvin-sign fold), so the isASCII guard is needed. The cost is a second scan of the word on a non-hot path. If you want to avoid it, a small asciiKeyword helper would do.

Suggestions (non-blocking)

  1. Hash duplication. kwHash (insert side) and asciiUpperHash (lookup side) are separate FNV loops. If one is changed without the other, lookups would silently miss. The equivalence test would catch it, but a one-line comment on kwHash saying it must match asciiUpperHash would help.
  2. kwHotKeys coverage. Hot keys that are missing from the maps are skipped silently. A test that every hot key resolves, ideally at displacement 0, would stop this list from going stale. It is cheap to add.
  3. Entry size. kwEntry is about 32 bytes (string 16, typ, three bools), so the table is about 16 KB. That is fine given the measured gains. Packing the flags would only matter if profiling points to cache misses.
  4. Compound second-word matching (around tokenizer.go:1163) still uses map lookups. That is a possible follow-up.

Tests

TestKwTableMatchesMaps and TestIsKeywordMatchesMap cover case variants, non-ASCII input, over-long words, duplicate keys and load factor. The token goldens are untouched. Coverage is adequate.

Security

No concerns. The table is built only from internal constants, and the key buffer is bounded by keywordKeyBufLen.

Performance

The CI numbers meet the plan thresholds (ParseE2E -4% to -7.5%, Tokenize/scratch -13% to -22%, instructions/op down 4.7% to 8.8%). The local join regression (+15.6%) did not reproduce on CI (-21.7%), so it was noise.
· branch perf/w2-01-keyword-table

@ajitpratap0 @claude
Merge origin/main into perf/w2-01-keyword-table
b6f9249 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
@github-actions

github-actions Bot commented Nov 1, 2026 •
edited
Loading

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 19s —— View job

Review of PR #920: fused-hash keyword table

Verdict: correct and low-risk, with no blockers. I read the full diff of keyword_lookup.go and tokenizer.go. I did not run the tests or benchmarks, so the performance numbers below come from the CI perf-gate.

Correctness

  • Table build. init() merges both maps into one key set. A key in both gets one slot with isKeyword and compoundStart merged. Only compoundKeywordStarts entries that are true are inserted.
  • Termination. init() panics if the key count exceeds kwTableSize/2, so the table is never full and the kwLookup probe loop always reaches an unused slot.
  • Deterministic layout. Hot keys go in first, then the rest in sorted order. Probe placement is the same on every run.
  • Case handling. asciiUpperHash hashes the upper-cased bytes, which matches how kwHash hashes the keys at insert time. Non-ASCII words return ok=false and take the old strings.ToUpper map path.
  • readIdentifier. typ is only used when isKeyword is true, so a compound-start-only entry behaves like the old map miss.
  • keywordAt. IsKeyword(word) && isASCII(word) keeps the old behaviour of rejecting non-ASCII words. IsKeyword can accept folds such as ſelect through strings.ToUpper, so the isASCII guard is needed.
  • Concurrency. The table is written only in init() and is read-only afterwards, so there is no race. e.key == string(key) does not allocate.

Suggestions (non-blocking)

  1. kwHash and asciiUpperHash. They are two copies of the same FNV loop. The equivalence test would catch drift, but a comment on kwHash saying it must match asciiUpperHash would help. Alternatively, kwHash could call asciiUpperHash on a stack buffer.
  2. keywordAt guard. Add a one-line comment explaining why isASCII is there. A test asserting that keywordAt rejects ſelect would also help.
  3. kwHotKeys staleness. Hot keys missing from the maps are skipped silently. TestKeywordTableHotKeysAtHome should cover this, but make sure it fails when a hot key is absent and does not skip it.
  4. Init order. init() reads the two maps. This is safe as long as both are plain package-level var initializers and not filled by another init(). TestKwTableMatchesMaps would catch a violation.
  5. Compound second-word lookup. tokenizer.go still uses asciiUpper and map lookups for the second word of a compound keyword. That could be a follow-up.
  6. Entry size. kwEntry is about 32 bytes, so the table is about 16 KB. That is fine given the measured gains. Packing the flags only matters if profiling points to cache misses.

Performance

The CI perf-gate (shadow mode) passes, with a suite geomean of -6.90% at p<0.001. ParseE2E is -4% to -8% on all 11 queries. Tokenize/scratch is -13% to -20%. allocs/op and B/op are unchanged, instructions/op fell 5% to 9%, and the A/A control is clean.

Tests

TestKwTableMatchesMaps and TestIsKeywordMatchesMap give good equivalence coverage. They cover case variants, non-ASCII input, over-long words, duplicate keys and load factor. The token goldens are untouched.

Security

No concerns. The table is built only from static data, and the stack buffers are bounded by keywordKeyBufLen.

The CHANGELOG entry is present.
· branch perf/w2-01-keyword-table

@ajitpratap0
ajitpratap0 merged commit 69a5a72 into main Nov 1, 2026
48 checks passed
@ajitpratap0
ajitpratap0 deleted the perf/w2-01-keyword-table branch November 1, 2026 17:26
@ajitpratap0 ajitpratap0 mentioned this pull request Nov 1, 2026
Merged

This branch was successfully deployed

1 active deployment
Preview — b6f9249e Deployed Nov 1, 2026 by vercel[bot]
View deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation tokenizer Tokenizer / lexer changes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ajitpratap0