Repository navigation
docs: re-measure published benchmarks at v1.17.0 - #933
Merged
Merged
Conversation
The weekly claims-drift check on main failed because metrics.json and benchmarks/compare/results/latest.json were measured on 2026-10-22, before the v1.17.0 tag. Both are re-measured at the tag so the published numbers describe the released version. - Parse stage (Apple M4, BenchmarkPerformanceBaseline, median of 20 runs, interleaved with v1.16.0 so ambient load hits both equally): SELECT 363 ns, INSERT 433, CTE 683, window 648, complex 1296. v1.16.0 reproduces its published figures within 1-2% in the same run (345, 400, 634, 631, 1230), so v1.17.0 is 2.8-8.1% slower (geomean +5.8%). - Competitor harness: bench-compare run 41533021592 on the v1.17.0 tag (EPYC 7763, Go 1.26.9). GoSQLX/Vitess geomean 1.10. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Claude finished @ajitpratap0's task in 28s —— View job Review of PR #933: re-measure published benchmarks at v1.17.0
Summary: This is a data-only change, and I found no blocking issues. I read the diff. I did not re-run the benchmarks or the generator, so the Test Plan results come from the PR description. What I checked
Minor, non-blocking notes
Verdict: Looks good to merge once CI, in particular the Claims Drift check, passes. |
github-actions
Bot
added
documentation
Perf gate: PASS (shadow mode: report only, this check never fails)Base
Rules: time regresses at >= +3% with p < 0.01 (Mann-Whitney U), or a suite geomean >= +1.5% (per-round geomeans, same test); allocs/op must not increase; B/op must not rise more than 2%; instructions/op must not rise more than 1.5%. An A/A control with more than 10% of benchmarks moving over 3% is noisy: re-run once, then inconclusive (neutral). Gated suite (GOGC=off, GOMAXPROCS=1)
A/A control: 0 of 50 benchmarks moved more than 3%. Instructions per operation (cmd/perfdriver)
Secondary metric: default GC, GOMAXPROCS=1 (informational)
Environmentbenchstat (primary metric) |
|
Claude finished @ajitpratap0's task in 13s —— View job Review of PR #933: re-measure published benchmarks at v1.17.0I read the diff against Findings: none blocking. This is a data-only change. What I checked
Minor, non-blocking notes
Verdict: Looks good to merge once the Claims Drift check passes. |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Main's weekly Claims Drift and Signals check failed on 2026-11-02. The two published measurements were taken on 2026-10-22, before the v1.17.0 tag (2026-10-31):
scripts/gen-readme-metrics/metrics.jsonbenchmarks/compare/results/latest.jsonBoth are now re-measured at the v1.17.0 tag, so README, llms.txt and the website describe the released version, not unreleased work on main.
Parse stage. Apple M4,
BenchmarkPerformanceBaseline, median of 20 runs. Each round ran the v1.16.0 binary and then the v1.17.0 binary, so ambient load affects both equally:Competitor harness. Data from
bench-comparerun 41533021592, dispatched on the v1.17.0 tag (commit 3655a37, EPYC 7763, Go 1.26.9). The tag push never triggered this run. The file was copied to bothbenchmarks/compare/results/andwebsite/src/data/competitor-benchmarks.json. GoSQLX/peer e2e geomean:Test Plan
go run ./scripts/gen-readme-metrics --check: ok. README was regenerated with the generator.go run ./scripts/gen-readme-metrics --claims --about "<repo About>": ok. This is the exact step that failed.npm run typecheckok,npm test45/45, eslint clean onsrc/data.Breaking Changes
None (published data only).
🤖 Generated with Claude Code
https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2