Skip to content

Add a codecov.yml so the project coverage status tolerates ordinary drift - #1066

Open
alongd wants to merge 1 commit into
mainfrom
codecov-project-threshold
Open

alongd wants to merge 1 commit into
mainfrom
codecov-project-threshold

Conversation

@alongd

@alongd alongd commented Oct 2, 2026

Copy link
Copy Markdown
Member

What

Adds a codecov.yml (23 lines, no code change):

  • coverage.status.project.default.threshold: 1%
  • coverage.status.patch.default.target: 80%

Why

ARC has no Codecov configuration at all — .coveragerc configures coverage.py's collection, not
Codecov's status checks. With no config, Codecov falls back to its built-in defaults, under which
the project status is target: auto with a 0% threshold: total coverage may never dip by any
amount relative to the base commit.

The consequence is that nearly every PR adding code covered below the repository average (~67%)
reports a failing check while every test job, CodeQL and the functional suite are green. Observed
on the current open set, all red on codecov/project alone:

PR status
#653 67.03% (-0.10%) → 67.05% (-0.08%) after adding a genuine unit test
#860, #1016, #1039, #1050, #1054, #1055, #1056, #1057 red on codecov/project, everything else green

On #653, codecov/patch simultaneously reported "Coverage not affected" — the gate that actually
judges whether the PR's new code is tested was already satisfied. A check that fails on
one-tenth of a percentage point trains reviewers to ignore it, which costs us the patch signal too.

What this does and does not change

  • Does: lets total coverage drift up to 1% without failing the status.
  • Does: keep an explicit patch target, so the coverage of a PR's own new or changed lines
    remains a gating signal. This is the stricter half of the config, not a loosening.
  • Does not: touch any test or source file, change .coveragerc, or alter what coverage is
    collected.
  • Does not: raise ARC's actual coverage. That is a separate and much larger piece of work.

Note that Codecov re-evaluates statuses on new uploads, so already-open PRs will pick this up on
their next CI run rather than retroactively.

With no Codecov config in the repo, the project status ran on Codecov's
defaults (target auto, threshold 0%): total coverage could never dip by any
amount, so almost every PR that added code reported a failing check while all
test jobs, CodeQL and the functional suite were green.

Set a 1% threshold on the project status, and give the patch status an explicit
80% target so the coverage of a PR's own new code remains the signal that gates.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The Codecov configuration is valid and matches the stated coverage policy.

Review effort: Balanced
Findings: None

What changed in this PR

Adds Codecov status configuration to tolerate minor overall coverage drift while preserving patch coverage enforcement.

Changes:

  • Allows project coverage to decrease by up to 1%.
  • Requires 80% coverage for changed lines.
  • Configures Codecov’s PR comment layout.
File Description
codecov.yml Defines project and patch coverage thresholds and comment layout.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@codecov

codecov Bot commented Oct 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 67.07%. Comparing base (a7c2666) to head (ceff564).

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #1066   +/-   ##
=======================================
  Coverage   67.07%   67.07%           
=======================================
  Files         123      123           
  Lines       43663    43663           
  Branches    11152    11152           
=======================================
+ Hits        29285    29288    +3     
+ Misses      11224    11219    -5     
- Partials     3154     3156    +2     
Flag Coverage Δ
functionaltests 67.07% <ø> (+<0.01%) ⬆️
unittests 67.07% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants