Skip to content

[codex] Preserve seeded file-order reproducibility when randomizing test runs#3463

Draft
bpc-oss wants to merge 2 commits into
avajs:mainfrom
bpc-oss:codex/pr-3458-seed-repro-fix
Draft

[codex] Preserve seeded file-order reproducibility when randomizing test runs#3463
bpc-oss wants to merge 2 commits into
avajs:mainfrom
bpc-oss:codex/pr-3458-seed-repro-fix

Conversation

@bpc-oss

@bpc-oss bpc-oss commented Jun 27, 2026

Copy link
Copy Markdown

Summary

This fixes a reproducibility bug in seeded test-file randomization.

When --randomize or --seed is used, file ordering should be determined by the explicit file set plus the seed. Before this change, AVA still applied failingTestsFirst() before shuffling files. Since that cache-based reordering changes the input array, the same seed could produce a different file order after previous failures updated node_modules/.cache/ava/failing-tests.json.

This patch:

  • skips failingTestsFirst() when randomization is active
  • adds an integration test proving the same seed yields the same file order even when the failed-test cache changes

Why

Issue #595 explicitly asks for randomization that remains reproducible for debugging. Hidden cache state should not change the order produced by a given seed.

Validation

  • npx xo lib/api.js test-tap/integration/assorted.js
  • $env:CI='1'; npx tap test-tap/integration/assorted.js
  • npx tap test-tap/runner.js
  • repo-native Linux rerun in node:24.15.0-bookworm with the repo extracted at /tmp/ava/ava:
    • npx tap test-tap/reporters/default.js
    • ./scripts/test.sh
    • ./scripts/ci.sh

Notes

  • This intentionally preserves reproducibility over failed-test prioritization while randomization is enabled.
  • Existing non-randomized runs continue to use failingTestsFirst().
  • Earlier local failures were environment false negatives caused by Windows checkout issues and a temp path that prevented reporter stack-frame filtering. The fresh Linux rerun above is the authoritative validation evidence for this patch state.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants