perf(tokenizer): inline current-line window for token locations [W2-02] by ajitpratap0 · Pull Request #924 · ajitpratap0/GoSQLX

ajitpratap0 · 2026-11-01T19:05:37Z

Summary

Plan W2-02 (TOK-2). tokenize now caches the current line's [curLS, nextLS) window and line index, so each token's Start and End location is a range check plus a subtraction instead of two calls to the non-inlinable toSQLPosition. Out-of-window offsets go to locSlow, which uses the existing lookup and reloads the window. Reset clears the window. The window relies on lineStarts being immutable during tokenize; prescan and the BOM adjustment both finish before the first lookup. Ported from the verified prototype verify-tokenizer/TOK-2.patch. No API change, and token positions are identical.

Test Plan

New pkg/sql/tokenizer/line_window_test.go:
- A randomized oracle test, built on independent location arithmetic, over LF, CRLF, tabs, multibyte words and BOM, on a pooled tokenizer reused across inputs.
- An explicit BOM first-line column test.
- A stale-window-after-reuse test.
- A comments/EOF/empty-input sanity test.
- These pass on the base commit too, which shows behaviour is preserved.
task check, task test:cbinding, go test -count=1 ./pkg/compatibility/... and staticcheck on ./pkg/sql/tokenizer/: all pass. The behaviour goldens are unchanged.
Local A/B on a shared M4, indicative only (task bench:ab BASE=origin/main BENCH='./pkg/sql/tokenizer=^BenchmarkTokenize$' ROUNDS=10):
- Time: geomean sec/op 638.8n to 583.8n, -8.61% (scratch and public). Significant on 9 of 22 cells; the rest are noisy under load.
- Allocs/op and B/op: unchanged (the geomean cannot be computed because the scratch allocs/op are all 0, and every sample is equal).
CI amd64 perf gate (shadow) on head 48e72dd: PASS, no regressions, A/A control clean (0 of 32 moved more than 3%).
- Gated geomean -2.36% (p<0.001); secondary metric (default GC) -1.88%.
- Tokenize/scratch (the target): join -6.37%, complex -7.05%, insert -4.79%, all p<0.001. Their geomean is about -6.1%, slightly short of the PLAN threshold of <= -7%; reported, not chased.
- ParseE2E -0.5% to -2.8%, ParseRW/medium -2.0% to -5.0%.
- instructions/op -4.5% to -7.6% on e2e cases.
- B/op unchanged; allocs/op unchanged everywhere.
- Full sticky comment on this PR.
Review follow-ups not taken: a locFast helper (the slow-path call would likely defeat inlining; the duplicated block is deliberate) and a white-box fast-path counter test.

Breaking Changes

None.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2

…ions [W2-02] Cache the current line's start offsets so the per-token Start and End locations are a range check and a subtraction instead of two calls to toSQLPosition. Out-of-window offsets use the existing lookup, which reloads the window. Reset clears it. Positions are unchanged; an oracle test covers LF/CRLF/tabs/multibyte/BOM and pooled reuse. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2

vercel · 2026-11-01T19:05:42Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
gosqlx-website	Ready	Preview	Nov 2, 2026 9:36am UTC

github-actions · 2026-11-01T19:06:25Z

Claude finished @ajitpratap0's task in 25s —— View job

Review of PR #924: inline current-line window for token locations

Verdict: looks correct and low-risk. I have only minor suggestions. I read the diff and compared it with toSQLPosition/lineLocation. I did not run the tests or benchmarks.

Correctness

Same results as the old path. Inside the window [curLS, nextLS), {curLi+1, idx-curLS+1} is what lineLocation returns. The min(pos, inputLen) clamp can't apply there, because the last line's nextLS = len(input)+1 makes idx == len(input) land inside the window without exceeding it.
Unsigned range check. uint(idx-curLS) < uint(nextLS-curLS) sends negative offsets and offsets past the end to locSlow, which clamps them as before. On a zeroed window (after Reset or before the first lookup) it is always false, so the first lookup takes the slow path. Both are safe.
locSlow and posCacheLine. It reads t.posCacheLine right after toSQLPosition sets it, so the window always matches the line just resolved. The li < len(t.lineStarts) guard covers empty lineStarts. Without it the window would stay zeroed and everything would take the slow path, which is still correct.
Window validity. It relies on lineStarts not changing during tokenize. The PR says prescan and the BOM adjustment finish first. Reset now clears the window, and the stale-after-reuse test covers that.
Other callers. nextToken and its helpers still call toSQLPosition and change posCacheLine, but they never touch the window, so the two stay independent.

Suggestions (non-blocking)

Duplicated inline block. The start and end sites repeat the same five lines. Go may not inline a small helper like func (t *Tokenizer) locFast(idx int) models.Location with the slow call in it. Check with -gcflags=-m. If it inlines, it removes the duplication at no cost. If not, a short comment at the second site pointing to the first would help.
Invariant comment. Add a comment next to the lineStarts rewrite sites, such as the BOM adjustment, saying the window must not be loaded before them. That would protect against a later refactor that moves a lookup earlier.
Tests. The oracle test is good. Two extra cases would help:
- A lone \r or an input ending in \n, so the final empty line lands exactly on idx == len(input) and the len(input)+1 bound is exercised.
- Tokens that span lines, such as multi-line strings or block comments, so the start and end are in different windows.
The PR says the new tests also pass on the base commit. That is what you want for a behaviour-preserving change, but it means they don't prove the new path is exercised. A cheap check is a counter-based or white-box test showing the fast path is taken for single-line input.
PR description. The perf-gate verdict is still PENDING, so fill it in before merge. The -8.61% geomean is local and indicative only, and only 9 of 22 cells are significant. The amd64 gate result should decide.
Changelog. The entry is clear. It could use the same [W2-02] tag placement as the neighbouring entries, which put it at the end before the final period. Cosmetic only.

Performance, security, race safety

No new allocations. locSlow is a cold path that runs about once per line.
No new input-dependent indexing: every lineStarts access is bounds-checked, and the fast path does no indexing.
The new fields are plain ints on a per-tokenizer instance, so there is no shared state and no race concern. Pool reuse resets them.

github-actions · 2026-11-01T19:20:17Z

Perf gate: PASS (shadow mode: report only, this check never fails)

Base 68a04b41e6a2, head cdf102059277. 10 interleaved rounds, benchtime 200ms, -cpu 1, pinned to CPU 3 (SMT siblings 2-3) with taskset.

9 benchmark(s) faster by 3% or more

Rules: time regresses at >= +3% with p < 0.01 (Mann-Whitney U), or a suite geomean >= +1.5% (per-round geomeans, same test); allocs/op must not increase; B/op must not rise more than 2%; instructions/op must not rise more than 1.5%. An A/A control with more than 10% of benchmarks moving over 3% is noisy: re-run once, then inconclusive (neutral).

Gated suite (GOGC=off, GOMAXPROCS=1)

benchmark	time	p	A/A	B/op	allocs/op	status
`ParseE2E/simple_select`	-1.10%	<0.001	+0.00%	+0.00%	5 → 5	ok
`ParseE2E/select_where`	-0.24%	0.140	+0.08%	+0.00%	18 → 18	ok
`ParseE2E/join`	-2.56%	<0.001	+0.09%	+0.00%	17 → 17	ok
`ParseE2E/aggregate`	-1.00%	0.005	+0.06%	+0.00%	18 → 18	ok
`ParseE2E/subquery`	-0.64%	0.009	+0.22%	+0.00%	14 → 14	ok
`ParseE2E/cte`	-2.15%	<0.001	+0.01%	+0.00%	21 → 21	ok
`ParseE2E/window`	-2.76%	<0.001	-0.15%	+0.00%	11 → 11	ok
`ParseE2E/complex`	-2.79%	<0.001	+0.10%	+0.00%	50 → 50	ok
`ParseE2E/insert`	-1.59%	<0.001	-0.19%	+0.00%	17 → 17	ok
`ParseE2E/update`	-1.78%	<0.001	+0.24%	+0.00%	13 → 13	ok
`ParseE2E/delete`	-0.24%	0.129	+0.00%	+0.00%	5 → 5	ok
`ParseRW/postgresql/medium`	-2.96%	<0.001	-0.40%	-0.00%	646 → 646	ok
`ParseRW/mysql/medium`	-3.29%	<0.001	-0.93%	+0.00%	365 → 365	faster
`ParseRW/mariadb/medium`	-4.20%	<0.001	+0.11%	-0.00%	523 → 523	faster
`ParseRW/sqlserver/medium`	-3.71%	<0.001	+0.62%	-0.00%	936 → 936	faster
`ParseRW/oracle/medium`	-3.03%	<0.001	+0.18%	-0.00%	1093 → 1093	faster
`ParseRW/sqlite/medium`	-1.51%	<0.001	-0.04%	+0.00%	346 → 346	ok
`ParseRW/snowflake/medium`	-3.97%	<0.001	-0.23%	+0.00%	777 → 777	faster
`ParseRW/clickhouse/medium`	-3.27%	<0.001	-0.27%	+0.00%	384 → 384	faster
`ParseFresh/simple_select`	+0.48%	0.345	+0.20%	+0.00%	6 → 6	ok
`ParseFresh/join`	-0.02%	0.971	+0.07%	+0.00%	18 → 18	ok
`ParseFresh/complex`	-0.42%	0.019	+0.41%	+0.00%	51 → 51	ok
`PoolGetPut/serial`	-1.86%	0.089	-0.23%	+0.00%	0 → 0	ok
`Tokenize/scratch/join`	-6.40%	<0.001	-0.02%	+0.00%	0 → 0	faster
`Tokenize/scratch/complex`	-7.06%	<0.001	-0.22%	+0.00%	0 → 0	faster
`Tokenize/scratch/insert`	-4.75%	<0.001	+0.00%	+0.00%	0 → 0	faster
`Normalize/simple_select`	+0.89%	0.005	+0.02%	+0.00%	10 → 10	ok
`Normalize/join`	-0.07%	0.384	+0.41%	+0.00%	40 → 40	ok
`Normalize/complex`	-1.10%	0.035	+0.10%	+0.00%	110 → 110	ok
`TransformParse/simple_select`	-1.18%	<0.001	+0.03%	+0.00%	5 → 5	ok
`TransformParse/join`	-1.66%	<0.001	-0.09%	+0.00%	17 → 17	ok
`TransformParse/complex`	-1.71%	<0.001	-0.06%	+0.00%	50 → 50	ok
geomean	-2.13%	<0.001	+0.00%

A/A control: 0 of 32 benchmarks moved more than 3%.

Instructions per operation (cmd/perfdriver)

case	instructions/op	status
`e2e/aggregate`	-6.15%	ok
`e2e/complex`	-7.65%	ok
`e2e/cte`	-5.83%	ok
`e2e/delete`	-4.61%	ok
`e2e/insert`	-6.92%	ok
`e2e/join`	-7.52%	ok
`e2e/select_where`	-5.59%	ok
`e2e/simple_select`	-4.48%	ok
`e2e/subquery`	-5.50%	ok
`e2e/update`	-6.18%	ok
`e2e/window`	-6.28%	ok
`rw/postgresql/medium`	-6.56%	ok

Secondary metric: default GC, GOMAXPROCS=1 (informational)

benchmark	time	p
`ParseE2E/simple_select`	-0.70%	0.002
`ParseE2E/select_where`	-0.33%	0.705
`ParseE2E/join`	-1.11%	0.009
`ParseE2E/aggregate`	-1.07%	0.070
`ParseE2E/subquery`	-1.41%	0.008
`ParseE2E/cte`	-1.92%	0.003
`ParseE2E/window`	-2.85%	0.009
`ParseE2E/complex`	-1.42%	<0.001
`ParseE2E/insert`	-2.01%	0.001
`ParseE2E/update`	-1.27%	0.082
`ParseE2E/delete`	-0.33%	0.677
`ParseRW/postgresql/medium`	-2.43%	<0.001
`ParseRW/mysql/medium`	-2.73%	<0.001
`ParseRW/mariadb/medium`	-1.93%	<0.001
`ParseRW/sqlserver/medium`	-2.19%	<0.001
`ParseRW/oracle/medium`	-1.32%	<0.001
`ParseRW/sqlite/medium`	-1.24%	0.002
`ParseRW/snowflake/medium`	-2.26%	<0.001
`ParseRW/clickhouse/medium`	-2.05%	<0.001
`ParseFresh/simple_select`	-0.37%	0.089
`ParseFresh/join`	-0.89%	0.063
`ParseFresh/complex`	-0.76%	0.162
`PoolGetPut/serial`	-2.05%	0.002
`Tokenize/scratch/join`	-6.83%	<0.001
`Tokenize/scratch/complex`	-7.42%	<0.001
`Tokenize/scratch/insert`	-4.75%	<0.001
`Normalize/simple_select`	+1.78%	<0.001
`Normalize/join`	-0.85%	0.089
`Normalize/complex`	-0.46%	0.190
`TransformParse/simple_select`	-0.99%	0.023
`TransformParse/join`	-1.36%	0.003
`TransformParse/complex`	-0.66%	0.011
geomean	-1.77%

Environment

date:        2026-11-02T09:37:04Z
base:        68a04b41e6a23b15ba97104a1f29aa5452ce98b1
head:        working tree of cdf1020592771b212a5a7acbb939107c7b7b2187
go:          go version go1.26.1 linux/amd64
kernel:      Linux 7.0.0-1017-azure x86_64
pinning:     pinned to CPU 3 (SMT siblings 2-3) with taskset
clocksource: tsc
load:        1.34, 0.54, 0.21
instr count: cachegrind
perf probe:  Error: No supported events found. The instructions:u event is not supported. 

Architecture:                            x86_64
CPU(s):                                  4
Model name:                              AMD EPYC 7763 64-Core Processor
Thread(s) per core:                      2
Core(s) per socket:                      2
Socket(s):                               1
Hypervisor vendor:                       Microsoft
L1d cache:                               64 KiB (2 instances)
L1i cache:                               64 KiB (2 instances)
L2 cache:                                1 MiB (2 instances)
L3 cache:                                32 MiB (1 instance)

benchstat (primary metric)

goos: linux
goarch: amd64
pkg: github.com/ajitpratap0/GoSQLX/pkg/fingerprint
cpu: AMD EPYC 7763 64-Core Processor                
                        │    base     │                head                │
                        │   sec/op    │   sec/op     vs base               │
Normalize/simple_select   2.853µ ± 0%   2.878µ ± 0%  +0.89% (p=0.003 n=10)
Normalize/join            10.58µ ± 1%   10.57µ ± 1%       ~ (p=0.382 n=10)
Normalize/complex         26.74µ ± 2%   26.45µ ± 2%       ~ (p=0.035 n=10)
geomean                   9.309µ        9.301µ       -0.09%

                        │     base     │                 head                  │
                        │     B/op     │     B/op      vs base                 │
Normalize/simple_select   1.016Ki ± 0%   1.016Ki ± 0%       ~ (p=1.000 n=10) ¹
Normalize/join            3.234Ki ± 0%   3.234Ki ± 0%       ~ (p=1.000 n=10) ¹
Normalize/complex         8.945Ki ± 0%   8.945Ki ± 0%       ~ (p=1.000 n=10) ¹
geomean                   3.086Ki        3.086Ki       +0.00%
¹ all samples are equal

                        │    base    │                head                 │
                        │ allocs/op  │ allocs/op   vs base                 │
Normalize/simple_select   10.00 ± 0%   10.00 ± 0%       ~ (p=1.000 n=10) ¹
Normalize/join            40.00 ± 0%   40.00 ± 0%       ~ (p=1.000 n=10) ¹
Normalize/complex         110.0 ± 0%   110.0 ± 0%       ~ (p=1.000 n=10) ¹
geomean                   35.30        35.30       +0.00%
¹ all samples are equal

pkg: github.com/ajitpratap0/GoSQLX/pkg/gosqlx
                          │    base     │                head                 │
                          │   sec/op    │    sec/op     vs base               │
ParseE2E/simple_select      1.817µ ± 1%    1.797µ ± 0%  -1.10% (p=0.000 n=10)
ParseE2E/select_where       3.828µ ± 1%    3.819µ ± 1%       ~ (p=0.138 n=10)
ParseE2E/join               5.790µ ± 1%    5.642µ ± 1%  -2.56% (p=0.000 n=10)
ParseE2E/aggregate          4.194µ ± 1%    4.152µ ± 1%  -1.00% (p=0.003 n=10)
ParseE2E/subquery           3.614µ ± 1%    3.591µ ± 1%  -0.64% (p=0.006 n=10)
ParseE2E/cte                5.046µ ± 1%    4.937µ ± 2%  -2.15% (p=0.001 n=10)
ParseE2E/window             3.302µ ± 1%    3.211µ ± 1%  -2.76% (p=0.000 n=10)
ParseE2E/complex            15.50µ ± 1%    15.07µ ± 1%  -2.79% (p=0.000 n=10)
ParseE2E/insert             2.583µ ± 1%    2.542µ ± 1%  -1.59% (p=0.000 n=10)
ParseE2E/update             2.338µ ± 1%    2.296µ ± 0%  -1.78% (p=0.000 n=10)
ParseE2E/delete             1.439µ ± 1%    1.436µ ± 1%       ~ (p=0.126 n=10)
ParseRW/postgresql/medium   199.2µ ± 3%    193.3µ ± 0%  -2.96% (p=0.000 n=10)
ParseRW/mysql/medium        110.2µ ± 2%    106.6µ ± 1%  -3.29% (p=0.000 n=10)
ParseRW/mariadb/medium      161.1µ ± 1%    154.3µ ± 1%  -4.20% (p=0.000 n=10)
ParseRW/sqlserver/medium    335.4µ ± 2%    323.0µ ± 0%  -3.71% (p=0.000 n=10)
ParseRW/oracle/medium       393.5µ ± 2%    381.5µ ± 1%  -3.03% (p=0.000 n=10)
ParseRW/sqlite/medium       92.65µ ± 2%    91.24µ ± 1%  -1.51% (p=0.000 n=10)
ParseRW/snowflake/medium    231.8µ ± 1%    222.6µ ± 0%  -3.97% (p=0.000 n=10)
ParseRW/clickhouse/medium   106.0µ ± 2%    102.6µ ± 1%  -3.27% (p=0.000 n=10)
ParseFresh/simple_select    9.995µ ± 1%   10.043µ ± 1%       ~ (p=0.342 n=10)
ParseFresh/join             23.27µ ± 1%    23.27µ ± 0%       ~ (p=0.971 n=10)
ParseFresh/complex          35.74µ ± 1%    35.59µ ± 0%       ~ (p=0.019 n=10)
PoolGetPut/serial           67.59n ± 1%    66.33n ± 3%       ~ (p=0.089 n=10)
geomean                     14.80µ         14.51µ       -1.95%

                          │     base     │                head                 │
                          │     B/s      │     B/s       vs base               │
ParseE2E/simple_select      13.65Mi ± 1%   13.80Mi ± 0%  +1.12% (p=0.000 n=10)
ParseE2E/select_where       25.66Mi ± 1%   25.73Mi ± 1%       ~ (p=0.137 n=10)
ParseE2E/join               20.75Mi ± 1%   21.30Mi ± 1%  +2.62% (p=0.000 n=10)
ParseE2E/aggregate          19.10Mi ± 1%   19.30Mi ± 1%  +1.02% (p=0.003 n=10)
ParseE2E/subquery           21.64Mi ± 1%   21.78Mi ± 1%  +0.62% (p=0.007 n=10)
ParseE2E/cte                23.44Mi ± 1%   23.95Mi ± 2%  +2.20% (p=0.001 n=10)
ParseE2E/window             27.16Mi ± 1%   27.92Mi ± 1%  +2.83% (p=0.000 n=10)
ParseE2E/complex            23.25Mi ± 1%   23.92Mi ± 1%  +2.87% (p=0.000 n=10)
ParseE2E/insert             28.06Mi ± 1%   28.51Mi ± 1%  +1.60% (p=0.000 n=10)
ParseE2E/update             22.44Mi ± 1%   22.85Mi ± 0%  +1.83% (p=0.000 n=10)
ParseE2E/delete             34.47Mi ± 1%   34.55Mi ± 1%       ~ (p=0.159 n=10)
ParseRW/postgresql/medium   18.01Mi ± 3%   18.55Mi ± 0%  +3.05% (p=0.000 n=10)
ParseRW/mysql/medium        24.56Mi ± 2%   25.40Mi ± 1%  +3.42% (p=0.000 n=10)
ParseRW/mariadb/medium      22.01Mi ± 1%   22.97Mi ± 1%  +4.35% (p=0.000 n=10)
ParseRW/sqlserver/medium    19.84Mi ± 2%   20.60Mi ± 0%  +3.85% (p=0.000 n=10)
ParseRW/oracle/medium       21.31Mi ± 2%   21.97Mi ± 1%  +3.13% (p=0.000 n=10)
ParseRW/sqlite/medium       22.30Mi ± 2%   22.64Mi ± 1%  +1.54% (p=0.000 n=10)
ParseRW/snowflake/medium    21.93Mi ± 1%   22.84Mi ± 0%  +4.13% (p=0.000 n=10)
ParseRW/clickhouse/medium   21.25Mi ± 2%   21.97Mi ± 1%  +3.39% (p=0.000 n=10)
geomean                     22.29Mi        22.81Mi       +2.31%

                          │      base      │                 head                  │
                          │      B/op      │     B/op      vs base                 │
ParseE2E/simple_select        816.0 ± 0%       816.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/select_where       1.523Ki ± 0%     1.523Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/join               2.305Ki ± 0%     2.305Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/aggregate          1.812Ki ± 0%     1.812Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/subquery           1.828Ki ± 0%     1.828Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/cte                2.336Ki ± 0%     2.336Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/window             1.281Ki ± 0%     1.281Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/complex            6.399Ki ± 0%     6.399Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/insert               824.0 ± 0%       824.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/update               736.0 ± 0%       736.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/delete               400.0 ± 0%       400.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/postgresql/medium   70.52Ki ± 0%     70.52Ki ± 0%  -0.00% (p=0.001 n=10)
ParseRW/mysql/medium        43.73Ki ± 0%     43.73Ki ± 0%       ~ (p=0.474 n=10)
ParseRW/mariadb/medium      64.85Ki ± 0%     64.85Ki ± 0%  -0.00% (p=0.000 n=10)
ParseRW/sqlserver/medium    90.61Ki ± 0%     90.60Ki ± 0%  -0.00% (p=0.000 n=10)
ParseRW/oracle/medium       113.8Ki ± 0%     113.8Ki ± 0%  -0.00% (p=0.000 n=10)
ParseRW/sqlite/medium       39.89Ki ± 0%     39.89Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/snowflake/medium    83.38Ki ± 0%     83.38Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/clickhouse/medium   37.13Ki ± 0%     37.13Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/simple_select      896.0 ± 0%       896.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/join             2.383Ki ± 0%     2.383Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/complex          6.477Ki ± 0%     6.477Ki ± 0%       ~ (p=1.000 n=10) ¹
PoolGetPut/serial             0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                                  ²                 -0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                          │     base      │                 head                 │
                          │   allocs/op   │  allocs/op   vs base                 │
ParseE2E/simple_select       5.000 ± 0%      5.000 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/select_where        18.00 ± 0%      18.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/join                17.00 ± 0%      17.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/aggregate           18.00 ± 0%      18.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/subquery            14.00 ± 0%      14.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/cte                 21.00 ± 0%      21.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/window              11.00 ± 0%      11.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/complex             50.00 ± 0%      50.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/insert              17.00 ± 0%      17.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/update              13.00 ± 0%      13.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/delete              5.000 ± 0%      5.000 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/postgresql/medium    646.0 ± 0%      646.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/mysql/medium         365.0 ± 0%      365.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/mariadb/medium       523.0 ± 0%      523.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/sqlserver/medium     936.0 ± 0%      936.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/oracle/medium       1.093k ± 0%     1.093k ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/sqlite/medium        346.0 ± 0%      346.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/snowflake/medium     777.0 ± 0%      777.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/clickhouse/medium    384.0 ± 0%      384.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/simple_select     6.000 ± 0%      6.000 ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/join              18.00 ± 0%      18.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/complex           51.00 ± 0%      51.00 ± 0%       ~ (p=1.000 n=10) ¹
PoolGetPut/serial            0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                                 ²                +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                          │    base     │                head                │
                          │  sec/stmt   │  sec/stmt    vs base               │
ParseRW/postgresql/medium   16.60µ ± 3%   16.11µ ± 0%  -2.96% (p=0.000 n=10)
ParseRW/mysql/medium        13.77µ ± 2%   13.32µ ± 1%  -3.29% (p=0.000 n=10)
ParseRW/mariadb/medium      17.90µ ± 1%   17.15µ ± 1%  -4.20% (p=0.000 n=10)
ParseRW/sqlserver/medium    15.97µ ± 2%   15.38µ ± 0%  -3.71% (p=0.000 n=10)
ParseRW/oracle/medium       16.39µ ± 2%   15.90µ ± 1%  -3.03% (p=0.000 n=10)
ParseRW/sqlite/medium       15.44µ ± 2%   15.21µ ± 1%  -1.51% (p=0.000 n=10)
ParseRW/snowflake/medium    17.83µ ± 1%   17.12µ ± 0%  -3.97% (p=0.000 n=10)
ParseRW/clickhouse/medium   15.15µ ± 2%   14.65µ ± 1%  -3.27% (p=0.000 n=10)
geomean                     16.08µ        15.56µ       -3.25%

pkg: github.com/ajitpratap0/GoSQLX/pkg/sql/tokenizer
                         │    base     │                head                │
                         │   sec/op    │   sec/op     vs base               │
Tokenize/scratch/join      2.631µ ± 1%   2.463µ ± 1%  -6.40% (p=0.000 n=10)
Tokenize/scratch/complex   7.116µ ± 1%   6.614µ ± 1%  -7.06% (p=0.000 n=10)
Tokenize/scratch/insert    1.262µ ± 0%   1.202µ ± 0%  -4.75% (p=0.000 n=10)
geomean                    2.869µ        2.695µ       -6.08%

                         │     base     │                head                 │
                         │     B/s      │     B/s       vs base               │
Tokenize/scratch/join      45.67Mi ± 1%   48.80Mi ± 1%  +6.85% (p=0.000 n=10)
Tokenize/scratch/complex   50.66Mi ± 1%   54.51Mi ± 1%  +7.61% (p=0.000 n=10)
Tokenize/scratch/insert    57.42Mi ± 0%   60.28Mi ± 0%  +4.98% (p=0.000 n=10)
geomean                    51.03Mi        54.33Mi       +6.47%

                         │     base     │                head                 │
                         │     B/op     │    B/op     vs base                 │
Tokenize/scratch/join      0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
Tokenize/scratch/complex   0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
Tokenize/scratch/insert    0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                               ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                         │     base     │                head                 │
                         │  allocs/op   │ allocs/op   vs base                 │
Tokenize/scratch/join      0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
Tokenize/scratch/complex   0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
Tokenize/scratch/insert    0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                               ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

pkg: github.com/ajitpratap0/GoSQLX/pkg/transform
                             │    base     │                head                │
                             │   sec/op    │   sec/op     vs base               │
TransformParse/simple_select   1.781µ ± 0%   1.760µ ± 0%  -1.18% (p=0.000 n=10)
TransformParse/join            6.233µ ± 1%   6.130µ ± 1%  -1.66% (p=0.000 n=10)
TransformParse/complex         16.47µ ± 1%   16.19µ ± 1%  -1.71% (p=0.000 n=10)
geomean                        5.676µ        5.589µ       -1.52%

                             │     base     │                 head                  │
                             │     B/op     │     B/op      vs base                 │
TransformParse/simple_select     816.0 ± 0%     816.0 ± 0%       ~ (p=1.000 n=10) ¹
TransformParse/join            2.305Ki ± 0%   2.305Ki ± 0%       ~ (p=1.000 n=10) ¹
TransformParse/complex         6.398Ki ± 0%   6.398Ki ± 0%       ~ (p=1.000 n=10) ¹
geomean                        2.273Ki        2.273Ki       +0.00%
¹ all samples are equal

                             │    base    │                head                 │
                             │ allocs/op  │ allocs/op   vs base                 │
TransformParse/simple_select   5.000 ± 0%   5.000 ± 0%       ~ (p=1.000 n=10) ¹
TransformParse/join            17.00 ± 0%   17.00 ± 0%       ~ (p=1.000 n=10) ¹
TransformParse/complex         50.00 ± 0%   50.00 ± 0%       ~ (p=1.000 n=10) ¹
geomean                        16.20        16.20       +0.00%
¹ all samples are equal

Run, raw data and benchstat

github-actions · 2026-11-02T09:36:23Z

Claude finished @ajitpratap0's task in 19s —— View job

Review of PR #924: inline current-line window for token locations

Verdict: correct and low-risk. I read the diff of tokenizer.go and pool.go. I did not run the tests or benchmarks, and I did not read line_window_test.go.

Correctness

Fast path. For curLS <= idx < nextLS it returns {curLi+1, idx-curLS+1}, which is what toSQLPosition returns inside that line.
Range check. The unsigned compare uint(idx-curLS) < uint(nextLS-curLS) sends negative and past-the-end offsets to locSlow. A zeroed window (after Reset, or before the first lookup) never matches, so the first lookup always takes the slow path.
Last line. nextLS = len(input)+1 lets idx == len(input) land in the window. That matches the old result for EOF tokens.
locSlow. It reads posCacheLine right after toSQLPosition sets it, so the window matches the line just resolved. The li < len(lineStarts) guard covers empty lineStarts; the window then stays zeroed and everything takes the slow path, which is still correct.
End location. The old code called getCurrentPosition() for End. The new code uses t.pos.Index directly. I didn't check that getCurrentPosition() simply wraps toSQLPosition(t.pos), so please confirm that.
Shared state. nextToken and its helpers still call toSQLPosition and move posCacheLine, but they never read the window, so the two caches stay independent.

Suggestions (non-blocking)

Invariant comment. The window assumes lineStarts is immutable. Add a short comment at the BOM adjustment and prescan sites saying no window lookup may happen before them. The field comment records this, but a later refactor would more likely trip over those sites.
Tests.
- Add an input ending in \n (or a lone \r), so EOF lands on the len(input)+1 bound.
- Add multi-line tokens (block comments, multi-line strings), so Start and End fall in different windows.
- The PR notes the new tests also pass on base. That's the right property for a behaviour-preserving change, but it means they don't show the fast path is taken. A white-box counter, or a check that curLS/nextLS are set after tokenizing, would cover that. The PR says you decided against this, which is reasonable.
Duplicated block. The start and end sites repeat the same check. Keeping it duplicated is fine if a helper would defeat inlining. Confirm with go build -gcflags=-m, and put a one-line "keep in sync" comment at the second site.
Perf. The CI amd64 gate passes. Tokenize/scratch is about -6.1% against the plan's ≤ -7% target; that's already noted in the PR. There are no allocation changes.

Security and races

The new fields are plain ints on a per-instance tokenizer, and Reset clears them on pool reuse. The fast path does no slice indexing, so it adds no out-of-bounds risk.

github-actions Bot added documentation Improvements or additions to documentation tokenizer Tokenizer / lexer changes labels Nov 1, 2026

vercel Bot deployed to Preview November 1, 2026 19:07

View deployment

Merge branch 'main' into perf/w2-02-line-window-positions

b9587e8

vercel Bot deployed to Preview November 2, 2026 09:36

View deployment

ajitpratap0 merged commit 5407cc0 into main Nov 2, 2026
55 of 56 checks passed

ajitpratap0 deleted the perf/w2-02-line-window-positions branch November 2, 2026 10:04

ajitpratap0 mentioned this pull request Nov 2, 2026

perf(tokenizer): zero-copy @param and @@var token values [W2-03] #925

Open

Navigation Menu

Repository navigation

perf(tokenizer): inline current-line window for token locations [W2-02] - #924

perf(tokenizer): inline current-line window for token locations [W2-02]#924
ajitpratap0 merged 2 commits into
mainfrom
perf/w2-02-line-window-positions

ajitpratap0 commented Nov 1, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Nov 1, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 1, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 1, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ajitpratap0 commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Plan

Breaking Changes

Uh oh!

vercel Bot commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review of PR #924: inline current-line window for token locations

Correctness

Suggestions (non-blocking)

Performance, security, race safety

Uh oh!

github-actions Bot commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Perf gate: PASS (shadow mode: report only, this check never fails)

Gated suite (GOGC=off, GOMAXPROCS=1)

Instructions per operation (cmd/perfdriver)

Uh oh!

github-actions Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review of PR #924: inline current-line window for token locations

Correctness

Suggestions (non-blocking)

Security and races

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ajitpratap0 commented Nov 1, 2026 •

edited

Loading

vercel Bot commented Nov 1, 2026 •

edited

Loading

github-actions Bot commented Nov 1, 2026 •

edited

Loading

github-actions Bot commented Nov 1, 2026 •

edited

Loading

github-actions Bot commented Nov 2, 2026 •

edited

Loading