Repository navigation
ci: rename the build comparison harness and let a pull request require no differences - #480
Merged
Merged
Conversation
"Harness" named a category of test code rather than what this tool does: compare the Quarto projects that two commits of great-docs write, along with the exit statuses, warnings, handled exceptions and calls to Quarto behind them. The package, its make targets and variable, its cache directory and its environment variables now carry the name `quarto_project_diff`. Base worktrees made under the old cache directory are no longer used; the first run creates a new one.
`--strict` said a comparison was stricter without saying how, and "strict" already means strict pyright in this repository. `--fail-on regression`, the default, fails only on regressions; `--fail-on difference` fails on any difference. The two values are the terms the report already uses, and `report.json` records the chosen one as `fail_on`. A comparison that matched in every case under the old `--strict` still ended its verdict with "but strict mode fails on any difference". A full match now reads only "MATCH".
Pull requests that change none of the package, the comparison tool, the Gauntlet packages, the project metadata, the Makefile or this workflow no longer install d2 and Quarto or build any package. The job still runs its lint and type checks and reports success. A pull request whose description holds the hidden marker `<!-- quarto-project-diff: fail-on difference -->` fails on any difference from its base commit, not only on regressions, so a change meant to leave every build alone is checked in CI. Editing the description re-runs the job only when the edit adds or removes the marker. The step name and the job summary state which mode ran.
`AGENTS.md` now lists `make quarto-project-diff`, which compares the Quarto projects a change writes with those from `main`, and tells agents to run it for changes to `great_docs/`. For a change that should leave every build unchanged, it tells them to run it with `--fail-on difference` and to put `<!-- quarto-project-diff: fail-on difference -->` in the pull request's description, so CI checks it the same way.
📚 Preview this PR's docs locallyFetch the just-built site and open it in your browser: # this exact build
great-docs preview --run 40952224183
# or always the latest build for this PR
great-docs preview --pr 480Auth & tips
|
Collaborator
Author
|
NOTE: @rich-iannone When you don't expect your PR to change the project that is rendered by quarto, add this to the PR body.
Building a selection of the gauntlet packages allows us to test a crucial chunk of the pipeline without paying the price of a full quarto render. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR renames the characterisation harness to the Quarto project diff (
make quarto-project-diff) and lets each pull request choose how strictly CI applies it. "Harness" named a kind of test code rather than what the tool does: compare the Quarto projects that two commits of great-docs write.Since #463, CI fails the comparison only on regressions, so a refactor that should leave every build unchanged was checked only by hand.
--strictbecomes--fail-on difference, and a pull request whose description ends with a hidden marker (seeAGENTS.md) runs it that way in CI. Pull requests that change nothing the comparison covers now skip it.