Skip to content

ci(test): shard the Race Detector job across four runners - #935

Merged
ajitpratap0 merged 1 commit into
mainfrom
ci/race-shards
Nov 3, 2026
Merged

ajitpratap0 merged 1 commit into
mainfrom
ci/race-shards

Conversation

@ajitpratap0

Copy link
Copy Markdown
Owner

Summary

  • The Race Detector job runs task test:race (every package except cbinding, -timeout 120s per package) on one 4-vCPU runner. The heavy race packages run side by side and starve each other, and their wall time swings with the runner. On identical main commits, tokenizer took 79–97s and security 65–103s; PR ci(perf): GOMEMLIMIT with GOGC=off, pinned toolchain, cachegrind I1, matrix mode [W0-15] #921, which touches only CI scripts, hit the tokenizer timeout at 120s (run 41586025728). test: parallelize the equivalence sweeps that push race tests past 120s #932 cut in-package time but cannot fix cross-package contention on a shared runner.
  • New scripts/race-shard.sh SHARD SHARDS deals packages round-robin, heaviest first (tokenizer, security, parser, transpiler, gosqlx, scripts/perf/gate), so tokenizer, security, parser and transpiler each land on their own shard. Packages, -race flags and the 120s timeout are unchanged.
  • New task test:race:shard SHARD=n SHARDS=4. task test:race is unchanged for local use and task check.
  • .github/workflows/test.yml: a four-way race-shard matrix plus a Race Detector aggregator job (needs: race-shard, if: always()). The aggregator passes only when all shards succeed, so the required status check name is kept.

Test Plan

  • Ran all four shards locally: 22 + 22 + 21 + 21 = 86 packages (the full test:race set), all pass.
  • Argument validation: race-shard.sh 4 4 exits 2; the task errors when SHARD or SHARDS is missing.
  • CI: four shard jobs plus the Race Detector aggregator pass on this PR.

Breaking Changes

None. CI and Taskfile only. The required check Race Detector keeps its name.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2

The race job runs every package on one 4-vCPU runner. The heavy race
packages (tokenizer, security, parser, transpiler) run side by side,
starve each other, and swing with the runner (tokenizer 79-120s,
security 65-104s on identical commits), so they cross the 120s
per-package timeout at random (PR #921, run 41586025728).

scripts/race-shard.sh deals the packages round-robin, heaviest first,
so each heavy package gets its own shard. Packages, flags and the 120s
timeout match `task test:race`. A `Race Detector` aggregator job keeps
the required check name and passes only if all four shards pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
@vercel

vercel Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
gosqlx-website Ignored Ignored Preview Nov 3, 2026 6:32am UTC

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 24s —— View job


Review of PR #935: shard the Race Detector job

I read the diff only. I did not run the shards or the workflow.

Verdict: Looks good. I found no blocking issues.

What works

  • Keeping the race-check job id and the Race Detector display name as an aggregator preserves the required status check. if: always() together with [ "$RESULT" = success ] is the right pattern. It also fails on cancelled and skipped, which is what you want.
  • fail-fast: false lets all shards report, so one flaky failure doesn't hide the others.
  • set -euo pipefail is safe with the ((cond)) && cmd lines. A failing && list that isn't the last command doesn't trip set -e, and each of those lines is followed by more work.
  • Argument validation is solid. It rejects non-numeric input, SHARDS=0 and SHARD>=SHARDS, and the requires: block in the Taskfile covers missing vars.
  • The empty-shard case exits 0 cleanly.
  • scripts/race-shard.sh ends with the same go test -race -timeout 120s as task test:race, so coverage and flags are unchanged.

Suggestions (non-blocking)

  1. Drift in the heavy list. The hardcoded heavy array will go stale as packages grow. The comment dates it, which helps. Consider a CI guard, for example a check that the union of all shards equals go list ./... | grep -v /cbinding. You already verified that locally (22+22+21+21 = 86), but nothing enforces it later. A package can't silently drop out as long as the script is the only partitioner, so this is low risk.
  2. Round-robin balance. The first six heavy packages land on shards 0, 1, 2, 3, 0, 1. So gosqlx shares a shard with tokenizer and scripts/perf/gate shares with security. If gosqlx is slow under race, those shards could still contend. If that shows up in CI timings, a snake order (0,1,2,3,3,2,1,0) or an explicit weight map would balance better.
  3. Cost. Each shard repeats checkout, Go setup and go install task. It's cheap with caching, and it's worth confirming the Go build cache is shared or restored per shard. Race builds are compile-heavy.
  4. Dependency on go list. go list ./... runs in every shard and needs a consistent module state. That's fine here. If go list ever prints build warnings to stdout, the package list would be polluted, though that is unlikely.
  5. Optional. The aggregator could print per-shard results with toJSON(needs) for easier debugging. This is a nicety only.

Test plan: The only unchecked item is the CI run on this PR. The four shard jobs and the aggregator should pass. Please also confirm that branch protection lists Race Detector and not the old job's path, since the shard jobs have new names.

@github-actions github-actions Bot added the ci-cd label Nov 3, 2026
@ajitpratap0
ajitpratap0 merged commit 9ccacb1 into main Nov 3, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant