feat(safegres): anon-role split, sealed runs, and an evaluation corpus - #1604
Merged
Conversation
An adapter already knows which role is the anonymous one -- Constructive's apis.anon_role, pgrst.db_anon_role, Supabase's anon, graphile's visitor -- and then flattened it into an undifferentiated exposure.roles. That left rolesFrom: 'exposure' unusable for the stacks whose signed-in role legitimately writes: pointing R1 at it on Constructive would flag every ordinary 'authenticated' grant as critical. - PlaneInput/ResolvedExposure/ExposureReport carry anonRoles, the subset of roles an unauthenticated caller arrives as; all four adapters populate it - rolesFrom takes 'anon' (default choice) or 'exposure' (the stricter reading); both still union with explicit roles - constructive, postgrest and graphile presets now resolve their anon role instead of naming it, so a custom anon_role is picked up rather than missed. The platform defaults stay listed explicitly, so an unresolved surface checks exactly what it checked before - pretty renderer marks which api roles are anonymous Behavior for existing configs is unchanged: a rule that names neither roles nor rolesFrom stays inert, and the shipped presets keep their previous role names.
…ummaries report.github.detail picks how much of the report goes to the job summary. GitHub truncates a summary at 1 MB, so a database with thousands of baselined findings had to hand-render its own compact markdown rather than use --github at all; 'summary' makes the first-party path usable there. The perf baseline verdict (N new / accepted / resolved) now renders at summary detail too: the absolute perf score is dominated by accepted debt, so the ratchet is the number a summary reader is actually looking for.
Contributor
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
A score is only meaningful against a known ruleset, and every knob that makes safegres useful in CI (rules, overrides, perf ignores, baselines, exposure) is also the cheapest way to raise a score without touching the database. That is fine when a team is declaring intent and fatal when the score is evaluating an agent. report.provenance now carries a sha256 over the *resolved* scoring surface, so it is invariant to how a posture was spelled and sensitive to anything that changes it, the version included. --sealed grades under a built-in preset alone and refuses the flags that would move the number; --verify-fingerprint lets a harness pin the ruler it expects.
~20 small schemas, each one deliberate flaw plus an answer key (expected findings, forbidden false positives, the fix). Cases are data, so a harness can consume them without running safegres. Two rule fixes the corpus surfaced: P1b only fired on VOLATILE SECURITY DEFINER functions, missing every STABLE wrapper (the common shape); and the recommended preset now points R1/R2/L5 at the anonymous role subset, so a declared exposure.anonRoles switches them on.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three pieces, in the order they depend on each other: the exposure surface learns which roles are anonymous, a run learns to state which ruler it was measured with, and the package ships schemas whose answers are known.
1.
anonRoles— the surface's missing distinction. The Constructive adapter reads bothrole_nameandanon_roleper API and flattened them into oneroleslist, sorolesFrom: 'exposure'would have pointed R1/R2 atauthenticatedand turned every ordinary signed-in write grant into a critical. A plane now carries both:Per adapter: Constructive
anon_roleonly; PostgRESTpgrst.db_anon_role(never the authenticator); Supabaseanonbut notauthenticated; Graphile's visitor role, which carries anonymous traffic by design.recommendednow points R1/R2/L5 atrolesFrom: 'anon', so a declaredexposure.anonRolesswitches them on and an undeclared surface leaves them inert exactly as before.2. Sealed runs. Every configuration knob is deliberate in CI and is the cheapest possible cheat when a score is what's being evaluated — turning off the rule and re-baselining both raise the number without touching the database. So every report now carries
provenance: { version, fingerprint, sealed, preset }, where the fingerprint is sha256 over the resolved score-relevant surface (rules × enabled × severity × options, overrides, scoring, public-read, perf ignores, exposure + adapter names, version) — invariant to how a posture was spelled, sensitive to anything that changes it.--sealedskips config discovery entirely and refuses--config/--rule/--exposure-schemas/baseline flags rather than ignoring them;--verify-fingerprint <f>exits non-zero unless the run matches the harness's expectation.3. The evaluation corpus (
corpus/). A sealed score says the ruler didn't move; it doesn't say the ruler is right. 23 small schemas, one deliberate flaw each, with an answer key as data (schema.sql+case.json: expected findings, forbidden false positives, worst severity, the one-sentence fix) so a harness can consume it without running safegres.loadCorpus()/gradeCase(report, case)grade recall and precision;__tests__/corpus.test.tsruns all of them and additionally pins that a flaw costs points exactly when the rule carries weight (infoand fail-closed rules are weightless by construction).Writing the corpus found two real bugs, both fixed here:
volatility === 'v'branch, so only a VOLATILESECURITY DEFINERfunction was reported — yet the rule is about inlining, and a STABLE definer wrapper is just as much a plan fence. Now checked before the volatility filter. This can surface new (medium, perf-dimension) findings on existing databases.infoseverities deduct nothing, which the corpus now asserts explicitly rather than leaving implicit.Also: configurable job-summary detail (
report.github.detail: 'summary' | 'normal' | 'verbose') — GitHub truncates a job summary at 1 MB, and a truncated report is worse than a short one, sosummarykeeps the badges and the ratchet verdict and drops the finding tables.Related: the deeper reachability work this unblocks is designed in constructive-io/constructive-planning#1361.
Link to Devin session: https://app.devin.ai/sessions/b7874ecee0c7471ea271e6e7193869dc
Requested by: @pyramation