Skip to content

feat(safegres): anon-role split, sealed runs, and an evaluation corpus - #1604

Merged
pyramation merged 4 commits into
mainfrom
feat/safegres-anon-roles
Aug 2, 2026
Merged

feat(safegres): anon-role split, sealed runs, and an evaluation corpus#1604
pyramation merged 4 commits into
mainfrom
feat/safegres-anon-roles

Conversation

@pyramation

@pyramation pyramation commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Three pieces, in the order they depend on each other: the exposure surface learns which roles are anonymous, a run learns to state which ruler it was measured with, and the package ships schemas whose answers are known.

1. anonRoles — the surface's missing distinction. The Constructive adapter reads both role_name and anon_role per API and flattened them into one roles list, so rolesFrom: 'exposure' would have pointed R1/R2 at authenticated and turned every ordinary signed-in write grant into a critical. A plane now carries both:

interface PlaneInput {
  roles?: string[];      // every role that reaches this plane
  anonRoles?: string[];  // the subset an *unauthenticated* caller arrives as
}
// rules choose: rolesFrom: 'anon' | 'exposure'

Per adapter: Constructive anon_role only; PostgREST pgrst.db_anon_role (never the authenticator); Supabase anon but not authenticated; Graphile's visitor role, which carries anonymous traffic by design. recommended now points R1/R2/L5 at rolesFrom: 'anon', so a declared exposure.anonRoles switches them on and an undeclared surface leaves them inert exactly as before.

2. Sealed runs. Every configuration knob is deliberate in CI and is the cheapest possible cheat when a score is what's being evaluated — turning off the rule and re-baselining both raise the number without touching the database. So every report now carries provenance: { version, fingerprint, sealed, preset }, where the fingerprint is sha256 over the resolved score-relevant surface (rules × enabled × severity × options, overrides, scoring, public-read, perf ignores, exposure + adapter names, version) — invariant to how a posture was spelled, sensitive to anything that changes it. --sealed skips config discovery entirely and refuses --config/--rule/--exposure-schemas/baseline flags rather than ignoring them; --verify-fingerprint <f> exits non-zero unless the run matches the harness's expectation.

3. The evaluation corpus (corpus/). A sealed score says the ruler didn't move; it doesn't say the ruler is right. 23 small schemas, one deliberate flaw each, with an answer key as data (schema.sql + case.json: expected findings, forbidden false positives, worst severity, the one-sentence fix) so a harness can consume it without running safegres. loadCorpus() / gradeCase(report, case) grade recall and precision; __tests__/corpus.test.ts runs all of them and additionally pins that a flaw costs points exactly when the rule carries weight (info and fail-closed rules are weightless by construction).

Writing the corpus found two real bugs, both fixed here:

  • P1b never fired on the common shape. It was emitted inside the volatility === 'v' branch, so only a VOLATILE SECURITY DEFINER function was reported — yet the rule is about inlining, and a STABLE definer wrapper is just as much a plan fence. Now checked before the volatility filter. This can surface new (medium, perf-dimension) findings on existing databases.
  • A5/A1 case authoring, not a bug but worth knowing: fail-closed rules and info severities deduct nothing, which the corpus now asserts explicitly rather than leaving implicit.

Also: configurable job-summary detail (report.github.detail: 'summary' | 'normal' | 'verbose') — GitHub truncates a job summary at 1 MB, and a truncated report is worse than a short one, so summary keeps the badges and the ratchet verdict and drops the finding tables.

Related: the deeper reachability work this unblocks is designed in constructive-io/constructive-planning#1361.

Link to Devin session: https://app.devin.ai/sessions/b7874ecee0c7471ea271e6e7193869dc
Requested by: @pyramation

An adapter already knows which role is the anonymous one -- Constructive's
apis.anon_role, pgrst.db_anon_role, Supabase's anon, graphile's visitor -- and
then flattened it into an undifferentiated exposure.roles. That left
rolesFrom: 'exposure' unusable for the stacks whose signed-in role legitimately
writes: pointing R1 at it on Constructive would flag every ordinary
'authenticated' grant as critical.

- PlaneInput/ResolvedExposure/ExposureReport carry anonRoles, the subset of
  roles an unauthenticated caller arrives as; all four adapters populate it
- rolesFrom takes 'anon' (default choice) or 'exposure' (the stricter reading);
  both still union with explicit roles
- constructive, postgrest and graphile presets now resolve their anon role
  instead of naming it, so a custom anon_role is picked up rather than missed.
  The platform defaults stay listed explicitly, so an unresolved surface checks
  exactly what it checked before
- pretty renderer marks which api roles are anonymous

Behavior for existing configs is unchanged: a rule that names neither roles nor
rolesFrom stays inert, and the shipped presets keep their previous role names.
…ummaries

report.github.detail picks how much of the report goes to the job summary.
GitHub truncates a summary at 1 MB, so a database with thousands of baselined
findings had to hand-render its own compact markdown rather than use --github
at all; 'summary' makes the first-party path usable there.

The perf baseline verdict (N new / accepted / resolved) now renders at summary
detail too: the absolute perf score is dominated by accepted debt, so the
ratchet is the number a summary reader is actually looking for.
@pyramation pyramation self-assigned this Aug 2, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

A score is only meaningful against a known ruleset, and every knob that makes
safegres useful in CI (rules, overrides, perf ignores, baselines, exposure) is
also the cheapest way to raise a score without touching the database. That is
fine when a team is declaring intent and fatal when the score is evaluating an
agent.

report.provenance now carries a sha256 over the *resolved* scoring surface, so
it is invariant to how a posture was spelled and sensitive to anything that
changes it, the version included. --sealed grades under a built-in preset alone
and refuses the flags that would move the number; --verify-fingerprint lets a
harness pin the ruler it expects.
~20 small schemas, each one deliberate flaw plus an answer key (expected findings, forbidden false positives, the fix). Cases are data, so a harness can consume them without running safegres.

Two rule fixes the corpus surfaced: P1b only fired on VOLATILE SECURITY DEFINER functions, missing every STABLE wrapper (the common shape); and the recommended preset now points R1/R2/L5 at the anonymous role subset, so a declared exposure.anonRoles switches them on.
@devin-ai-integration devin-ai-integration Bot changed the title feat(safegres): split anon roles out of the exposure surface feat(safegres): anon-role split, sealed runs, and an evaluation corpus Aug 2, 2026
@pyramation
pyramation merged commit e817b1a into main Aug 2, 2026
16 checks passed
@pyramation
pyramation deleted the feat/safegres-anon-roles branch August 2, 2026 04:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant