Skip to content

perf(ast): O(1) ReleaseAST for parser-built trees with no pooled draws [W4-01] - #930

Merged
ajitpratap0 merged 2 commits into
mainfrom
perf/w4-01-o1-release-ast
Nov 2, 2026
Merged

perf(ast): O(1) ReleaseAST for parser-built trees with no pooled draws [W4-01]#930
ajitpratap0 merged 2 commits into
mainfrom
perf/w4-01-o1-release-ast

Conversation

@ajitpratap0

@ajitpratap0 ajitpratap0 commented Nov 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Summary

W4-01 (PLAN D3 option b). The parser counts nodes it draws from the AST pools and reports the count to ast.ReleaseAST through an internal hook (pkg/sql/internal/astrelease, importable only under pkg/sql). A parser-built tree with zero pooled draws is released in O(1): only Statements references are cleared and the container is pooled. Trees with pooled draws (tuples, arrays, subscripts/slices, ALTER/SHOW/DESCRIBE/REPLACE, sequences) and all hand-built trees keep the full walk. No exported API change; one unexported AST.noPooledDraws field.

Notes:

  • 13 draw sites in pkg/sql/parser call p.notePooledDraw(). The heap-built &ast.TupleExpression{} in parseAssignmentTarget (UPDATE SET (a,b)=...) is counted too, so it is still recycled exactly as before.
  • Guard: TestReleaseDrawnPolicy_ParserCountsDraws fails if a parser function draws a pooled node without counting it; releaseDrawnConstructors guard unchanged.
  • CRITIQUE Postgres positional parameters support #17 caveat holds: INSERT multi-row VALUES does not draw tuples (verified), but (a,b) IN (...), UPDATE tuple assignment and PostgreSQL arrays keep the walk. Over the corpus x 5 dialects 10874 of 11774 parses are draw-free.

Test Plan

  • New: TestPooledDraws_CountMatchesTree (all 11 pooled types covered), TestPooledDraws_CorpusEquivalence (differential: no pooled node in any zero-draw tree, 11774 parses), TestPooledDraws_ResetPerParse, ast fast-path/hand-built/flag-reset tests.
  • task check pass, task test:cbinding pass, go test -count=1 ./pkg/compatibility/... pass, staticcheck on pkg/sql/ast, parser, internal clean. Behaviour goldens unchanged.
  • Local task bench:ab BASE=origin/main BENCH='./pkg/gosqlx=^BenchmarkParseE2E$' ROUNDS=8 (M4, shared, indicative): geomean -3.74% (p<0.001); simple -3.2, where -4.4, join -4.3, aggregate -6.4, subquery -3.4, cte -4.4, window -1.5, complex -4.3, insert -0.6, update -3.4, delete -5.2. B/op and allocs/op unchanged.
  • PLAN W4-01 target 1, BenchmarkParseStage/release <= -70% (interleaved fixed-iteration A/B on the M4, 6 rounds, -benchtime 200000x -cpu 1, base = origin/main afec16a; fixed Nx because task bench:ab never finishes on this benchmark under GOGC=off: the StopTimer batch calibration grows b.N into the millions): geomean -83.9% (p=0.002), B/op flat. Per query: select_where -83.7%, join -89.0%, aggregate -88.6%, subquery -88.4%, cte -89.1%, window -81.3%, complex -95.9%, update -74.9%, delete -71.4%. Two queries MISS the -70% bar: simple_select -66.1% and insert -61.1%. Reported, not hidden; the geomean clears the target.
  • Added TestPooledDraws_ParseMarksTree: real parses through ParseFromModelTokens and ParseContextFromModelTokens must mark the tree (noPooledDraws true for draw-free, false for a tuple, counter resets across parses). Removing the MarkParserBuilt call fails it.
  • Doc: ReleaseAST fast-path comments reworded: the skipped walk would Put only heap-built nodes into pools the parser never draws from (e.g. DROP/CREATE VIEW); the flag is a parse-time snapshot.
  • CI amd64 perf-gate (shadow) verdict: PASS. Geomean -3.70% (p<0.001), 24 benchmarks faster by >=3%, no regression, B/op and allocs/op unchanged, instructions/op -2% to -6.5%; ParseE2E geomean target (<= -2%) met (aggregate -6.2% vs C0 projection -3 to -4%). The other PLAN target, BenchmarkParseStage/release <= -70%, is reported separately below. Full report:

Perf gate: PASS (shadow mode: report only, this check never fails)

Base afec16a41252, head d8977d30714d. 10 interleaved rounds, benchtime 200ms, -cpu 1, pinned to CPU 3 (SMT siblings 2-3) with taskset.

  • 24 benchmark(s) faster by 3% or more

Rules: time regresses at >= +3% with p < 0.01 (Mann-Whitney U), or a suite geomean >= +1.5% (per-round geomeans, same test); allocs/op must not increase; B/op must not rise more than 2%; instructions/op must not rise more than 1.5%. An A/A control with more than 10% of benchmarks moving over 3% is noisy: re-run once, then inconclusive (neutral).

Gated suite (GOGC=off, GOMAXPROCS=1)

benchmark time p A/A B/op allocs/op status
ParseE2E/simple_select -4.49% <0.001 -0.10% +0.00% 5 → 5 faster
ParseE2E/select_where -3.49% <0.001 -0.02% +0.00% 18 → 18 faster
ParseE2E/join -3.94% <0.001 -0.36% +0.00% 17 → 17 faster
ParseE2E/aggregate -6.20% <0.001 -1.02% +0.00% 18 → 18 faster
ParseE2E/subquery -6.49% <0.001 -0.40% +0.00% 14 → 14 faster
ParseE2E/cte -6.28% <0.001 -0.47% +0.00% 21 → 21 faster
ParseE2E/window -3.68% <0.001 -0.56% +0.00% 11 → 11 faster
ParseE2E/complex -3.86% <0.001 -0.03% +0.00% 50 → 50 faster
ParseE2E/insert -0.57% 0.034 -0.13% +0.00% 17 → 17 ok
ParseE2E/update -3.10% <0.001 -0.07% +0.00% 13 → 13 faster
ParseE2E/delete -4.70% 0.002 -0.30% +0.00% 5 → 5 faster
ParseRW/postgresql/medium -3.43% <0.001 -0.15% -0.00% 646 → 646 faster
ParseRW/mysql/medium -3.51% <0.001 -0.27% -0.00% 365 → 365 faster
ParseRW/mariadb/medium -3.79% <0.001 -0.05% -0.00% 523 → 523 faster
ParseRW/sqlserver/medium -5.30% <0.001 +0.22% -0.05% 928 → 928 faster
ParseRW/oracle/medium -3.74% <0.001 +0.45% -0.00% 1093 → 1093 faster
ParseRW/sqlite/medium -3.53% <0.001 -0.20% +0.00% 346 → 346 faster
ParseRW/snowflake/medium -5.52% <0.001 -0.17% -0.00% 777 → 777 faster
ParseRW/clickhouse/medium -3.09% <0.001 -0.12% +0.00% 384 → 384 faster
ParseFresh/simple_select -2.13% <0.001 +0.02% +0.00% 6 → 6 ok
ParseFresh/join -2.79% <0.001 +0.14% +0.00% 18 → 18 ok
ParseFresh/complex -9.76% <0.001 +0.05% +0.00% 51 → 51 faster
PoolGetPut/serial -0.37% 0.273 -0.58% +0.00% 0 → 0 ok
Tokenize/scratch/join -0.84% 0.012 -0.51% +0.00% 0 → 0 ok
Tokenize/scratch/complex -0.19% 0.364 +0.30% +0.00% 0 → 0 ok
Tokenize/scratch/insert -0.32% 0.426 -0.05% +0.00% 0 → 0 ok
Normalize/simple_select -4.30% 0.003 +0.00% +0.00% 7 → 7 faster
Normalize/join -3.27% 0.003 -0.31% +0.00% 23 → 23 faster
Normalize/complex -2.53% <0.001 -0.10% +0.00% 69 → 69 ok
TransformParse/simple_select -5.03% <0.001 -0.24% +0.00% 5 → 5 faster
TransformParse/join -3.73% <0.001 +0.07% +0.00% 17 → 17 faster
TransformParse/complex -3.76% <0.001 +0.06% +0.00% 50 → 50 faster
geomean -3.70% <0.001 -0.15%

A/A control: 0 of 32 benchmarks moved more than 3%.

Instructions per operation (cmd/perfdriver)

case instructions/op status
e2e/aggregate -6.03% ok
e2e/complex -4.07% ok
e2e/cte -4.87% ok
e2e/delete -4.35% ok
e2e/insert -1.96% ok
e2e/join -3.63% ok
e2e/select_where -3.85% ok
e2e/simple_select -3.98% ok
e2e/subquery -6.47% ok
e2e/update -3.52% ok
e2e/window -3.88% ok
rw/postgresql/medium -3.26% ok

Breaking Changes

None.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2

@ajitpratap0 @claude
perf(ast): O(1) ReleaseAST for parser-built trees with no pooled draw…
07a0707 
…s [W4-01]

The parser counts nodes drawn from the AST pools and reports the count to
ast.ReleaseAST through an internal hook; a zero count skips the release walk.
Hand-built trees and trees with pooled draws keep the full walk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
@vercel

vercel Bot commented Nov 2, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
gosqlx-website Ready Ready Preview Nov 2, 2026 4:05pm UTC

@github-actions github-actions Bot added documentation Improvements or additions to documentation parser ast AST node changes labels Nov 2, 2026
@github-actions

github-actions Bot commented Nov 2, 2026 •
edited
Loading

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 39s —— View job

Review of PR #930: O(1) ReleaseAST for draw-free parser trees

I reviewed the diff for ast, internal/astrelease and the parser draw sites. I did not run the tests or benchmarks. This is a read-only review, and I made no changes.

Overall: The design is sound and conservative. The fast path is opt-in and set only by the parser. It fails safe: any tree that doesn't go through MarkParserBuilt, such as hand-built trees or other parse entry points, keeps the full walk. The internal package keeps the exported API unchanged.

What looks good

  • noPooledDraws defaults to false, so a missed or forgotten path degrades to the old behaviour.
  • ReleaseAST clears the flag before pooling, so a reused container can't inherit a stale "draw-free" mark.
  • Counting the heap-built &ast.TupleExpression{} in parseAssignmentTarget is a good catch. The release walk recycles that type, so it has to be counted to keep recycling it.
  • p.pooledDraws is reset in Reset(), parseTokens and parseContextTokens.
  • The guard test TestReleaseDrawnPolicy_ParserCountsDraws is the right safeguard against future draw sites being missed. The differential corpus test (11774 parses) also gives good coverage.

Points to consider

  1. Mutation after parse (ast.go flag, pool.go fast path). The flag is a snapshot taken at parse time. If a caller or package such as transform mutates a parsed tree and attaches pooled nodes, for example by calling ast.GetArraySubscriptExpression() and splicing it in, the fast path skips them.
    • The result is only a missed Put. The nodes go to the GC, with no correctness or race issue, so it is safe.
    • It does mean the "would Put nothing" comment holds only for unmodified trees. Worth a sentence in the noPooledDraws doc, or a check that transform and gosqlx don't splice pooled nodes into parsed trees.
  2. Counting at the draw site is manual. p.notePooledDraw() placement varies between before and after the Get* call, and parseBracketArrayLiteral has an odd blank line after it (expressions_complex.go, around the second hunk). That's cosmetic, but a tiny helper such as p.drawArray() wrapping Get and the count would make misses structurally impossible. The guard test covers this today, so it's optional.
  3. Error paths. MarkParserBuilt is called only on success, which is correct because no tree is returned on error. Pooled nodes drawn in a failed parse were never released before either, so there's no regression. Please confirm that any recovery-style parse API (if one exists) returning a partial AST either calls the hook with the right count or intentionally keeps the full walk.
  4. Package-init coupling (astrelease.go). The no-op default plus the init() assignment in ast is fine. Since parser imports ast, the init always runs before use. A one-line test asserting MarkParserBuilt is non-default after importing the parser would protect against the hook being silently dropped by a refactor. The ast fast-path tests may already cover this.
  5. clear(ast.Statements). This is equivalent to the old nil-ing loop and requires Go 1.21+, which is fine for 1.26. Note that it clears only len, as before.
  6. CHANGELOG and docs. The entry is present. The "CI amd64 perf-gate verdict: PENDING" item in the PR body should be filled in before merge, since the local geomean of -3.74% was measured on a shared M4 machine.

Security / concurrency

  • No security concerns.
  • The flag is written only before the tree is returned and read only in ReleaseAST by the owner, so there are no new races. task test:race is reported passing.

Verdict

Looks good to merge once the amd64 perf-gate result is in. The suggestions above are non-blocking, with the exception of confirming item 3.

@github-actions

github-actions Bot commented Nov 2, 2026 •
edited
Loading

Copy link
Copy Markdown

Perf gate: PASS (shadow mode: report only, this check never fails)

Base afec16a41252, head 033b4ff8e852. 10 interleaved rounds, benchtime 200ms, -cpu 1, pinned to CPU 3 (SMT siblings 2-3) with taskset.

  • 23 benchmark(s) faster by 3% or more

Rules: time regresses at >= +3% with p < 0.01 (Mann-Whitney U), or a suite geomean >= +1.5% (per-round geomeans, same test); allocs/op must not increase; B/op must not rise more than 2%; instructions/op must not rise more than 1.5%. An A/A control with more than 10% of benchmarks moving over 3% is noisy: re-run once, then inconclusive (neutral).

Gated suite (GOGC=off, GOMAXPROCS=1)

benchmark time p A/A B/op allocs/op status
ParseE2E/simple_select -4.26% <0.001 -0.14% +0.00% 5 → 5 faster
ParseE2E/select_where -3.30% <0.001 +0.11% +0.00% 18 → 18 faster
ParseE2E/join -3.64% <0.001 -0.12% +0.00% 17 → 17 faster
ParseE2E/aggregate -5.42% <0.001 -0.07% +0.00% 18 → 18 faster
ParseE2E/subquery -5.95% <0.001 +0.13% +0.00% 14 → 14 faster
ParseE2E/cte -5.68% <0.001 +0.03% +0.00% 21 → 21 faster
ParseE2E/window -3.47% <0.001 +0.20% +0.00% 11 → 11 faster
ParseE2E/complex -3.98% <0.001 -0.24% +0.00% 50 → 50 faster
ParseE2E/insert -0.45% 0.003 -0.04% +0.00% 17 → 17 ok
ParseE2E/update -3.48% <0.001 -0.18% +0.00% 13 → 13 faster
ParseE2E/delete -4.74% <0.001 -0.13% +0.00% 5 → 5 faster
ParseRW/postgresql/medium -3.13% <0.001 +0.13% -0.00% 646 → 646 faster
ParseRW/mysql/medium -3.30% <0.001 +0.18% -0.00% 365 → 365 faster
ParseRW/mariadb/medium -3.91% <0.001 -0.09% -0.00% 523 → 523 faster
ParseRW/sqlserver/medium -5.53% <0.001 -0.28% -0.05% 928 → 928 faster
ParseRW/oracle/medium -4.29% <0.001 -0.24% -0.00% 1093 → 1093 faster
ParseRW/sqlite/medium -3.77% 0.002 -0.42% +0.00% 346 → 346 faster
ParseRW/snowflake/medium -5.39% <0.001 -0.19% -0.00% 777 → 777 faster
ParseRW/clickhouse/medium -2.93% <0.001 -0.17% +0.00% 384 → 384 ok
ParseFresh/simple_select -1.80% 0.001 -0.05% +0.00% 6 → 6 ok
ParseFresh/join -2.33% 0.002 -0.13% +0.00% 18 → 18 ok
ParseFresh/complex -6.41% <0.001 +3.70% +0.00% 51 → 51 faster
PoolGetPut/serial +0.18% 0.005 +0.04% +0.00% 0 → 0 ok
Tokenize/scratch/join -0.32% 0.058 +0.07% +0.00% 0 → 0 ok
Tokenize/scratch/complex -0.21% 0.850 -0.06% +0.00% 0 → 0 ok
Tokenize/scratch/insert +0.05% 0.789 -0.09% +0.00% 0 → 0 ok
Normalize/simple_select -4.37% <0.001 +0.00% +0.00% 7 → 7 faster
Normalize/join -3.01% <0.001 +0.03% +0.00% 23 → 23 faster
Normalize/complex -2.40% <0.001 -0.03% +0.00% 69 → 69 ok
TransformParse/simple_select -4.87% <0.001 -0.04% +0.00% 5 → 5 faster
TransformParse/join -3.59% <0.001 +0.40% +0.00% 17 → 17 faster
TransformParse/complex -3.69% <0.001 -0.12% +0.00% 50 → 50 faster
geomean -3.43% <0.001 +0.07%

A/A control: 1 of 32 benchmarks moved more than 3%.

Instructions per operation (cmd/perfdriver)

case instructions/op status
e2e/aggregate -6.06% ok
e2e/complex -4.12% ok
e2e/cte -4.83% ok
e2e/delete -4.28% ok
e2e/insert -1.95% ok
e2e/join -3.62% ok
e2e/select_where -3.87% ok
e2e/simple_select -4.07% ok
e2e/subquery -6.49% ok
e2e/update -3.48% ok
e2e/window -3.84% ok
rw/postgresql/medium -3.25% ok
Secondary metric: default GC, GOMAXPROCS=1 (informational)
benchmark time p
ParseE2E/simple_select -4.53% <0.001
ParseE2E/select_where -4.85% <0.001
ParseE2E/join -2.33% <0.001
ParseE2E/aggregate -7.06% <0.001
ParseE2E/subquery -6.84% <0.001
ParseE2E/cte -5.82% <0.001
ParseE2E/window -3.69% <0.001
ParseE2E/complex -5.87% <0.001
ParseE2E/insert -1.16% 0.026
ParseE2E/update -3.45% <0.001
ParseE2E/delete -5.70% <0.001
ParseRW/postgresql/medium -5.25% <0.001
ParseRW/mysql/medium -5.15% <0.001
ParseRW/mariadb/medium -5.81% <0.001
ParseRW/sqlserver/medium -5.56% <0.001
ParseRW/oracle/medium -5.39% <0.001
ParseRW/sqlite/medium -4.91% <0.001
ParseRW/snowflake/medium -6.44% <0.001
ParseRW/clickhouse/medium -3.53% <0.001
ParseFresh/simple_select -1.61% <0.001
ParseFresh/join -1.71% <0.001
ParseFresh/complex -2.84% <0.001
PoolGetPut/serial +0.10% 0.014
Tokenize/scratch/join -0.36% 0.009
Tokenize/scratch/complex -0.19% 0.041
Tokenize/scratch/insert -0.14% 0.025
Normalize/simple_select -4.71% <0.001
Normalize/join -3.73% 0.005
Normalize/complex -3.82% 0.002
TransformParse/simple_select -5.40% <0.001
TransformParse/join -1.94% <0.001
TransformParse/complex -5.31% <0.001
geomean -3.93%
Environment 
date:        2026-11-02T16:05:41Z
base:        afec16a412525e69a3db8ef8c6edea88f3636454
head:        working tree of 033b4ff8e852112d2efbc67122eb3ea3aac12c0d
go:          go version go1.26.1 linux/amd64
kernel:      Linux 7.0.0-1017-azure x86_64
pinning:     pinned to CPU 3 (SMT siblings 2-3) with taskset
clocksource: tsc
load:        2.20, 0.63, 0.22
instr count: cachegrind
perf probe:  Error: No supported events found. The instructions:u event is not supported. 

Architecture:                            x86_64
CPU(s):                                  4
Model name:                              INTEL(R) XEON(R) PLATINUM 8573C
Thread(s) per core:                      2
Core(s) per socket:                      2
Socket(s):                               1
CPU(s) scaling MHz:                      134%
Hypervisor vendor:                       Microsoft
L1d cache:                               96 KiB (2 instances)
L1i cache:                               64 KiB (2 instances)
L2 cache:                                4 MiB (2 instances)
L3 cache:                                260 MiB (1 instance)
benchstat (primary metric) 
goos: linux
goarch: amd64
pkg: github.com/ajitpratap0/GoSQLX/pkg/fingerprint
cpu: INTEL(R) XEON(R) PLATINUM 8573C
                        │    base     │                head                │
                        │   sec/op    │   sec/op     vs base               │
Normalize/simple_select   2.069µ ± 0%   1.979µ ± 0%  -4.37% (p=0.000 n=10)
Normalize/join            7.350µ ± 0%   7.128µ ± 0%  -3.01% (p=0.000 n=10)
Normalize/complex         18.49µ ± 0%   18.05µ ± 0%  -2.40% (p=0.000 n=10)
geomean                   6.551µ        6.337µ       -3.27%

                        │     base     │                 head                  │
                        │     B/op     │     B/op      vs base                 │
Normalize/simple_select     896.0 ± 0%     896.0 ± 0%       ~ (p=1.000 n=10) ¹
Normalize/join            2.680Ki ± 0%   2.680Ki ± 0%       ~ (p=1.000 n=10) ¹
Normalize/complex         7.492Ki ± 0%   7.492Ki ± 0%       ~ (p=1.000 n=10) ¹
geomean                   2.600Ki        2.600Ki       +0.00%
¹ all samples are equal

                        │    base    │                head                 │
                        │ allocs/op  │ allocs/op   vs base                 │
Normalize/simple_select   7.000 ± 0%   7.000 ± 0%       ~ (p=1.000 n=10) ¹
Normalize/join            23.00 ± 0%   23.00 ± 0%       ~ (p=1.000 n=10) ¹
Normalize/complex         69.00 ± 0%   69.00 ± 0%       ~ (p=1.000 n=10) ¹
geomean                   22.31        22.31       +0.00%
¹ all samples are equal

pkg: github.com/ajitpratap0/GoSQLX/pkg/gosqlx
                          │    base     │                head                │
                          │   sec/op    │   sec/op     vs base               │
ParseE2E/simple_select      1.421µ ± 1%   1.360µ ± 1%  -4.26% (p=0.000 n=10)
ParseE2E/select_where       3.224µ ± 0%   3.118µ ± 0%  -3.30% (p=0.000 n=10)
ParseE2E/join               4.896µ ± 1%   4.718µ ± 0%  -3.64% (p=0.000 n=10)
ParseE2E/aggregate          3.618µ ± 1%   3.422µ ± 1%  -5.42% (p=0.000 n=10)
ParseE2E/subquery           3.035µ ± 1%   2.855µ ± 1%  -5.95% (p=0.000 n=10)
ParseE2E/cte                4.295µ ± 1%   4.051µ ± 4%  -5.68% (p=0.000 n=10)
ParseE2E/window             2.721µ ± 1%   2.626µ ± 0%  -3.47% (p=0.000 n=10)
ParseE2E/complex            13.10µ ± 0%   12.58µ ± 1%  -3.98% (p=0.000 n=10)
ParseE2E/insert             2.247µ ± 1%   2.237µ ± 0%  -0.45% (p=0.001 n=10)
ParseE2E/update             1.984µ ± 1%   1.915µ ± 3%  -3.48% (p=0.000 n=10)
ParseE2E/delete             1.140µ ± 1%   1.086µ ± 1%  -4.74% (p=0.000 n=10)
ParseRW/postgresql/medium   153.6µ ± 0%   148.8µ ± 0%  -3.13% (p=0.000 n=10)
ParseRW/mysql/medium        84.25µ ± 0%   81.47µ ± 0%  -3.30% (p=0.000 n=10)
ParseRW/mariadb/medium      124.9µ ± 1%   120.0µ ± 2%  -3.91% (p=0.000 n=10)
ParseRW/sqlserver/medium    272.4µ ± 0%   257.4µ ± 1%  -5.53% (p=0.000 n=10)
ParseRW/oracle/medium       329.8µ ± 0%   315.7µ ± 0%  -4.29% (p=0.000 n=10)
ParseRW/sqlite/medium       73.32µ ± 1%   70.56µ ± 2%  -3.77% (p=0.002 n=10)
ParseRW/snowflake/medium    187.1µ ± 1%   177.0µ ± 1%  -5.39% (p=0.000 n=10)
ParseRW/clickhouse/medium   81.71µ ± 0%   79.32µ ± 0%  -2.93% (p=0.000 n=10)
ParseFresh/simple_select    6.328µ ± 0%   6.214µ ± 0%  -1.80% (p=0.000 n=10)
ParseFresh/join             10.43µ ± 0%   10.19µ ± 0%  -2.33% (p=0.001 n=10)
ParseFresh/complex          19.34µ ± 4%   18.11µ ± 1%  -6.41% (p=0.000 n=10)
PoolGetPut/serial           93.97n ± 0%   94.14n ± 0%  +0.18% (p=0.003 n=10)
geomean                     11.70µ        11.26µ       -3.79%

                          │     base     │                head                 │
                          │     B/s      │     B/s       vs base               │
ParseE2E/simple_select      17.46Mi ± 1%   18.23Mi ± 1%  +4.43% (p=0.000 n=10)
ParseE2E/select_where       30.47Mi ± 0%   31.51Mi ± 0%  +3.41% (p=0.000 n=10)
ParseE2E/join               24.55Mi ± 1%   25.47Mi ± 0%  +3.77% (p=0.000 n=10)
ParseE2E/aggregate          22.14Mi ± 1%   23.41Mi ± 1%  +5.73% (p=0.000 n=10)
ParseE2E/subquery           25.77Mi ± 1%   27.39Mi ± 1%  +6.31% (p=0.000 n=10)
ParseE2E/cte                27.54Mi ± 1%   29.20Mi ± 4%  +6.04% (p=0.000 n=10)
ParseE2E/window             32.95Mi ± 1%   34.13Mi ± 0%  +3.59% (p=0.000 n=10)
ParseE2E/complex            27.52Mi ± 0%   28.66Mi ± 1%  +4.14% (p=0.000 n=10)
ParseE2E/insert             32.26Mi ± 1%   32.40Mi ± 0%  +0.44% (p=0.001 n=10)
ParseE2E/update             26.44Mi ± 1%   27.39Mi ± 3%  +3.61% (p=0.000 n=10)
ParseE2E/delete             43.52Mi ± 1%   45.67Mi ± 1%  +4.95% (p=0.000 n=10)
ParseRW/postgresql/medium   23.36Mi ± 0%   24.11Mi ± 0%  +3.25% (p=0.000 n=10)
ParseRW/mysql/medium        32.12Mi ± 0%   33.22Mi ± 0%  +3.41% (p=0.000 n=10)
ParseRW/mariadb/medium      28.40Mi ± 1%   29.55Mi ± 2%  +4.06% (p=0.000 n=10)
ParseRW/sqlserver/medium    24.42Mi ± 0%   25.85Mi ± 1%  +5.86% (p=0.000 n=10)
ParseRW/oracle/medium       25.42Mi ± 0%   26.56Mi ± 0%  +4.50% (p=0.000 n=10)
ParseRW/sqlite/medium       28.17Mi ± 1%   29.28Mi ± 2%  +3.93% (p=0.001 n=10)
ParseRW/snowflake/medium    27.18Mi ± 1%   28.72Mi ± 1%  +5.68% (p=0.000 n=10)
ParseRW/clickhouse/medium   27.58Mi ± 0%   28.41Mi ± 0%  +3.01% (p=0.000 n=10)
geomean                     27.29Mi        28.44Mi       +4.21%

                          │      base      │                 head                  │
                          │      B/op      │     B/op      vs base                 │
ParseE2E/simple_select        816.0 ± 0%       816.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/select_where       1.523Ki ± 0%     1.523Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/join               2.305Ki ± 0%     2.305Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/aggregate          1.812Ki ± 0%     1.812Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/subquery           1.828Ki ± 0%     1.828Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/cte                2.336Ki ± 0%     2.336Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/window             1.281Ki ± 0%     1.281Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/complex            6.399Ki ± 0%     6.399Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/insert               824.0 ± 0%       824.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/update               736.0 ± 0%       736.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/delete               400.0 ± 0%       400.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/postgresql/medium   70.51Ki ± 0%     70.51Ki ± 0%  -0.00% (p=0.001 n=10)
ParseRW/mysql/medium        43.73Ki ± 0%     43.72Ki ± 0%  -0.00% (p=0.000 n=10)
ParseRW/mariadb/medium      64.84Ki ± 0%     64.84Ki ± 0%  -0.00% (p=0.002 n=10)
ParseRW/sqlserver/medium    90.53Ki ± 0%     90.49Ki ± 0%  -0.05% (p=0.000 n=10)
ParseRW/oracle/medium       113.8Ki ± 0%     113.8Ki ± 0%  -0.00% (p=0.000 n=10)
ParseRW/sqlite/medium       39.89Ki ± 0%     39.89Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/snowflake/medium    83.38Ki ± 0%     83.38Ki ± 0%  -0.00% (p=0.000 n=10)
ParseRW/clickhouse/medium   37.13Ki ± 0%     37.13Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/simple_select      896.0 ± 0%       896.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/join             2.383Ki ± 0%     2.383Ki ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/complex          6.477Ki ± 0%     6.477Ki ± 0%       ~ (p=1.000 n=10) ¹
PoolGetPut/serial             0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                                  ²                 -0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                          │     base      │                 head                 │
                          │   allocs/op   │  allocs/op   vs base                 │
ParseE2E/simple_select       5.000 ± 0%      5.000 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/select_where        18.00 ± 0%      18.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/join                17.00 ± 0%      17.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/aggregate           18.00 ± 0%      18.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/subquery            14.00 ± 0%      14.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/cte                 21.00 ± 0%      21.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/window              11.00 ± 0%      11.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/complex             50.00 ± 0%      50.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/insert              17.00 ± 0%      17.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/update              13.00 ± 0%      13.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseE2E/delete              5.000 ± 0%      5.000 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/postgresql/medium    646.0 ± 0%      646.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/mysql/medium         365.0 ± 0%      365.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/mariadb/medium       523.0 ± 0%      523.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/sqlserver/medium     928.0 ± 0%      928.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/oracle/medium       1.093k ± 0%     1.093k ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/sqlite/medium        346.0 ± 0%      346.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/snowflake/medium     777.0 ± 0%      777.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseRW/clickhouse/medium    384.0 ± 0%      384.0 ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/simple_select     6.000 ± 0%      6.000 ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/join              18.00 ± 0%      18.00 ± 0%       ~ (p=1.000 n=10) ¹
ParseFresh/complex           51.00 ± 0%      51.00 ± 0%       ~ (p=1.000 n=10) ¹
PoolGetPut/serial            0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                                 ²                +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                          │    base     │                head                │
                          │  sec/stmt   │  sec/stmt    vs base               │
ParseRW/postgresql/medium   12.80µ ± 0%   12.40µ ± 0%  -3.13% (p=0.000 n=10)
ParseRW/mysql/medium        10.53µ ± 0%   10.18µ ± 0%  -3.29% (p=0.000 n=10)
ParseRW/mariadb/medium      13.87µ ± 1%   13.33µ ± 2%  -3.91% (p=0.000 n=10)
ParseRW/sqlserver/medium    12.97µ ± 0%   12.26µ ± 1%  -5.53% (p=0.000 n=10)
ParseRW/oracle/medium       13.74µ ± 0%   13.15µ ± 0%  -4.29% (p=0.000 n=10)
ParseRW/sqlite/medium       12.22µ ± 1%   11.76µ ± 2%  -3.77% (p=0.002 n=10)
ParseRW/snowflake/medium    14.39µ ± 1%   13.61µ ± 1%  -5.39% (p=0.000 n=10)
ParseRW/clickhouse/medium   11.67µ ± 0%   11.33µ ± 0%  -2.93% (p=0.000 n=10)
geomean                     12.72µ        12.20µ       -4.03%

pkg: github.com/ajitpratap0/GoSQLX/pkg/sql/tokenizer
                         │    base     │                head                │
                         │   sec/op    │   sec/op     vs base               │
Tokenize/scratch/join      2.202µ ± 0%   2.195µ ± 1%       ~ (p=0.054 n=10)
Tokenize/scratch/complex   5.848µ ± 0%   5.836µ ± 2%       ~ (p=0.836 n=10)
Tokenize/scratch/insert    1.056µ ± 0%   1.056µ ± 1%       ~ (p=0.780 n=10)
geomean                    2.386µ        2.383µ       -0.16%

                         │     base     │                head                 │
                         │     B/s      │     B/s       vs base               │
Tokenize/scratch/join      54.56Mi ± 0%   54.73Mi ± 1%       ~ (p=0.055 n=10)
Tokenize/scratch/complex   61.65Mi ± 0%   61.78Mi ± 2%       ~ (p=0.809 n=10)
Tokenize/scratch/insert    68.67Mi ± 0%   68.65Mi ± 1%       ~ (p=0.869 n=10)
geomean                    61.36Mi        61.46Mi       +0.16%

                         │     base     │                head                 │
                         │     B/op     │    B/op     vs base                 │
Tokenize/scratch/join      0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
Tokenize/scratch/complex   0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
Tokenize/scratch/insert    0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                               ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                         │     base     │                head                 │
                         │  allocs/op   │ allocs/op   vs base                 │
Tokenize/scratch/join      0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
Tokenize/scratch/complex   0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
Tokenize/scratch/insert    0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                               ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

pkg: github.com/ajitpratap0/GoSQLX/pkg/transform
                             │    base     │                head                │
                             │   sec/op    │   sec/op     vs base               │
TransformParse/simple_select   1.417µ ± 0%   1.348µ ± 0%  -4.87% (p=0.000 n=10)
TransformParse/join            5.341µ ± 0%   5.150µ ± 1%  -3.59% (p=0.000 n=10)
TransformParse/complex         14.18µ ± 0%   13.66µ ± 0%  -3.69% (p=0.000 n=10)
geomean                        4.752µ        4.559µ       -4.05%

                             │     base     │                 head                  │
                             │     B/op     │     B/op      vs base                 │
TransformParse/simple_select     816.0 ± 0%     816.0 ± 0%       ~ (p=1.000 n=10) ¹
TransformParse/join            2.305Ki ± 0%   2.305Ki ± 0%       ~ (p=1.000 n=10) ¹
TransformParse/complex         6.398Ki ± 0%   6.398Ki ± 0%       ~ (p=1.000 n=10) ¹
geomean                        2.273Ki        2.273Ki       +0.00%
¹ all samples are equal

                             │    base    │                head                 │
                             │ allocs/op  │ allocs/op   vs base                 │
TransformParse/simple_select   5.000 ± 0%   5.000 ± 0%       ~ (p=1.000 n=10) ¹
TransformParse/join            17.00 ± 0%   17.00 ± 0%       ~ (p=1.000 n=10) ¹
TransformParse/complex         50.00 ± 0%   50.00 ± 0%       ~ (p=1.000 n=10) ¹
geomean                        16.20        16.20       +0.00%
¹ all samples are equal

Run, raw data and benchstat

@ajitpratap0 @claude
test(parser): assert real parses mark trees for O(1) release; fix fas…
c3dbf10 
…t-path docs [W4-01]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
@github-actions

github-actions Bot commented Nov 2, 2026 •
edited
Loading

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 27s —— View job

Review of PR #930: O(1) ReleaseAST for draw-free parser trees

I read the diff for ast, internal/astrelease and parser.go. I did not re-run the tests or benchmarks. I took the test and perf results from the PR body and the CI report, and I did not review every draw-site hunk in detail.

Overall: The design is sound and fails safe. The flag defaults to false, only the parser sets it, and only on success. ReleaseAST clears the flag before the container is pooled, so a reused container can't inherit a stale mark. The perf gate passes: geomean -3.70%, with B/op and allocs/op unchanged.

Points to consider (none blocking)

  1. The flag is a parse-time snapshot. If a caller splices a pooled node into a parsed tree later, for example via ast.GetArraySubscriptExpression(), the fast path skips it. That node only loses recycling and goes to the GC, so there is no correctness or race issue. The ReleaseAST doc comment already says this. The noPooledDraws field comment in ast.go could say it too.
  2. Manual counting at draw sites. A missed notePooledDraw() call silently turns into a missed Put. TestReleaseDrawnPolicy_ParserCountsDraws and the corpus differential test cover this today. A small wrapper that does the Get* and the count together would make a miss impossible by construction, but this is optional.
  3. Other parse entry points. MarkParserBuilt is called in parseTokens and parseContextTokens, and pooledDraws is reset in both and in Reset(). Please confirm that no other path returns an *ast.AST, such as a recovery or partial-parse API. If one exists, it should either call the hook or deliberately keep the full walk. It is safe either way, because the flag is false by default.
  4. Hook wiring. astrelease.MarkParserBuilt is a no-op until ast's init() runs. This is fine, since the parser imports ast. A test that fails if the fast path never triggers for a parsed tree would catch a refactor that drops the hook. The new ast fast-path and TestPooledDraws_* tests probably cover this already.
  5. clear(ast.Statements). This is equivalent to the old nil-ing loop and fine on Go 1.26. It clears len elements only, as before.
  6. Panics mid-walk. Nothing changes here. The fast path is strictly less work than the old walk.

Security / concurrency

No new concerns. The flag is written before the tree is returned and read only by the owner in ReleaseAST. task test:race is reported passing in the PR body.

Test coverage

Coverage looks good. The new tests are the count-vs-tree check over all 11 pooled types, a differential corpus check (11,774 parses), a per-parse reset test, and the ast fast-path, hand-built and flag-reset tests.

Verdict

Looks good to merge. The perf gate is in and meets the plan's thresholds. Item 3 is worth a quick check, and the rest are optional polish.

@ajitpratap0
ajitpratap0 merged commit 533148d into main Nov 2, 2026
51 of 52 checks passed
@ajitpratap0
ajitpratap0 deleted the perf/w4-01-o1-release-ast branch November 2, 2026 16:32
ajitpratap0 added a commit that referenced this pull request Nov 2, 2026
@ajitpratap0 @claude
Merge remote-tracking branch 'origin/main' into perf/w4-04-coalloc-sm…
1a94b9c 
…all-slices

Brings in #926 (tokens built in place) and #930 (O(1) ReleaseAST).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
@ajitpratap0 ajitpratap0 mentioned this pull request Nov 3, 2026
Open

This branch was successfully deployed

1 active deployment
Preview — c3dbf107 Deployed Nov 2, 2026 by vercel[bot]
View deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

ast AST node changes documentation Improvements or additions to documentation parser

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ajitpratap0