-
Notifications
You must be signed in to change notification settings - Fork 1
Add verified-backport evidence contract #694
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
justin808
wants to merge
6
commits into
main
Choose a base branch
from
jg-codex/issue-204-verified-backport-core
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
21d52a3
Add verified backport evidence classifier
justin808 6b256b1
Fix verified backport evidence validation
justin808 f8ac848
Add explicit usage error for verified backport classify
justin808 84cf4c1
Distinguish input read failures in verified backport classify
justin808 0b73626
Merge remote-tracking branch 'origin/main' into HEAD
justin808 2430534
Fix verified-backport UTF-8 and policy classification
justin808 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,270 @@ | ||
| { | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "$id": "https://shakacode.github.io/agent-workflows/schemas/verified-backport-v1.json", | ||
| "title": "Verified Backport Evidence v1", | ||
| "description": "Evidence for deciding whether a backport is mechanically patch-equivalent. Classification never satisfies consumer target-branch policy, branch protection, or current-head gates.", | ||
| "type": "object", | ||
| "required": [ | ||
| "schema", | ||
| "source", | ||
| "source_evidence", | ||
| "target", | ||
| "patch", | ||
| "target_only_delta", | ||
| "reused_evidence", | ||
| "target_requirements", | ||
| "review_generated_changes", | ||
| "forward_port_dispositions" | ||
| ], | ||
| "properties": { | ||
| "schema": {"const": "verified-backport-v1"}, | ||
| "source": {"$ref": "#/$defs/source"}, | ||
| "source_evidence": {"$ref": "#/$defs/sourceEvidence"}, | ||
| "target": {"$ref": "#/$defs/target"}, | ||
| "patch": {"$ref": "#/$defs/patch"}, | ||
| "target_only_delta": {"$ref": "#/$defs/targetOnlyDelta"}, | ||
| "reused_evidence": { | ||
| "type": "array", | ||
| "uniqueItems": true, | ||
| "items": {"$ref": "#/$defs/reusedEvidence"} | ||
| }, | ||
| "target_requirements": {"$ref": "#/$defs/targetRequirements"}, | ||
| "review_generated_changes": { | ||
| "type": "array", | ||
| "uniqueItems": true, | ||
| "items": {"$ref": "#/$defs/reviewGeneratedChange"} | ||
| }, | ||
| "forward_port_dispositions": { | ||
| "type": "array", | ||
| "uniqueItems": true, | ||
| "items": {"$ref": "#/$defs/forwardPortDisposition"} | ||
| } | ||
| }, | ||
| "additionalProperties": false, | ||
| "$defs": { | ||
| "nonemptyString": {"type": "string", "minLength": 1}, | ||
| "nonemptyStringOrUnknown": { | ||
| "anyOf": [ | ||
| {"$ref": "#/$defs/nonemptyString"}, | ||
| {"const": "UNKNOWN"} | ||
| ] | ||
| }, | ||
| "durableUrlOrUnknown": { | ||
| "oneOf": [ | ||
| {"type": "string", "pattern": "^https://[^\\s]+$"}, | ||
| {"const": "UNKNOWN"} | ||
| ] | ||
| }, | ||
| "sha": {"type": "string", "pattern": "^[0-9a-f]{40}$"}, | ||
| "shaOrUnknown": { | ||
| "oneOf": [ | ||
| {"$ref": "#/$defs/sha"}, | ||
| {"const": "UNKNOWN"} | ||
| ] | ||
| }, | ||
| "patchIdOrUnknown": { | ||
| "oneOf": [ | ||
| {"type": "string", "pattern": "^[0-9a-f]{40}$"}, | ||
| {"const": "UNKNOWN"} | ||
| ] | ||
| }, | ||
| "stringListOrUnknown": { | ||
| "oneOf": [ | ||
| { | ||
| "type": "array", | ||
| "uniqueItems": true, | ||
| "items": {"$ref": "#/$defs/nonemptyString"} | ||
| }, | ||
| {"const": "UNKNOWN"} | ||
| ] | ||
| }, | ||
| "nonemptyStringListOrUnknown": { | ||
| "oneOf": [ | ||
| { | ||
| "type": "array", | ||
| "minItems": 1, | ||
| "uniqueItems": true, | ||
| "items": {"$ref": "#/$defs/nonemptyString"} | ||
| }, | ||
| {"const": "UNKNOWN"} | ||
| ] | ||
| }, | ||
| "source": { | ||
| "type": "object", | ||
| "required": ["repository", "pull_request", "author", "head_sha", "merge_sha", "trusted_status"], | ||
| "properties": { | ||
| "repository": { | ||
| "oneOf": [ | ||
| {"type": "string", "pattern": "^[^/\\s]+/[^/\\s]+$"}, | ||
| {"const": "UNKNOWN"} | ||
| ] | ||
| }, | ||
| "pull_request": { | ||
| "oneOf": [ | ||
| {"type": "integer", "minimum": 1}, | ||
| {"const": "UNKNOWN"} | ||
| ] | ||
| }, | ||
| "author": {"$ref": "#/$defs/nonemptyStringOrUnknown"}, | ||
| "head_sha": {"$ref": "#/$defs/shaOrUnknown"}, | ||
| "merge_sha": {"$ref": "#/$defs/shaOrUnknown"}, | ||
| "trusted_status": {"enum": ["merged", "untrusted", "UNKNOWN"]} | ||
| }, | ||
| "additionalProperties": false | ||
| }, | ||
| "sourceEvidence": { | ||
| "type": "object", | ||
| "required": ["required_coverage", "reviews", "checks"], | ||
| "properties": { | ||
| "required_coverage": {"$ref": "#/$defs/requiredCoverage"}, | ||
| "reviews": { | ||
| "type": "array", | ||
| "items": {"$ref": "#/$defs/reviewEvidence"} | ||
| }, | ||
| "checks": { | ||
| "type": "array", | ||
| "items": {"$ref": "#/$defs/checkEvidence"} | ||
| } | ||
| }, | ||
| "additionalProperties": false | ||
| }, | ||
| "requiredCoverage": { | ||
| "type": "object", | ||
| "required": ["policy_source", "review_ids", "check_ids"], | ||
| "properties": { | ||
| "policy_source": {"$ref": "#/$defs/nonemptyStringOrUnknown"}, | ||
| "review_ids": {"$ref": "#/$defs/nonemptyStringListOrUnknown"}, | ||
| "check_ids": {"$ref": "#/$defs/nonemptyStringListOrUnknown"} | ||
| }, | ||
| "additionalProperties": false | ||
| }, | ||
| "reviewEvidence": { | ||
| "type": "object", | ||
| "required": ["id", "actor", "head_sha", "status", "url"], | ||
| "properties": { | ||
| "id": {"$ref": "#/$defs/nonemptyStringOrUnknown"}, | ||
| "actor": {"$ref": "#/$defs/nonemptyStringOrUnknown"}, | ||
| "head_sha": {"$ref": "#/$defs/shaOrUnknown"}, | ||
| "status": {"enum": ["accepted", "rejected", "pending", "UNKNOWN"]}, | ||
| "url": {"$ref": "#/$defs/durableUrlOrUnknown"} | ||
| }, | ||
| "additionalProperties": false | ||
| }, | ||
| "checkEvidence": { | ||
| "type": "object", | ||
| "required": ["id", "name", "head_sha", "status", "url"], | ||
| "properties": { | ||
| "id": {"$ref": "#/$defs/nonemptyStringOrUnknown"}, | ||
| "name": {"$ref": "#/$defs/nonemptyStringOrUnknown"}, | ||
| "head_sha": {"$ref": "#/$defs/shaOrUnknown"}, | ||
| "status": {"enum": ["passed", "failed", "pending", "UNKNOWN"]}, | ||
| "url": {"$ref": "#/$defs/durableUrlOrUnknown"} | ||
| }, | ||
| "additionalProperties": false | ||
| }, | ||
| "target": { | ||
| "type": "object", | ||
| "required": ["branch", "base_sha", "head_sha"], | ||
| "properties": { | ||
| "branch": {"$ref": "#/$defs/nonemptyStringOrUnknown"}, | ||
| "base_sha": {"$ref": "#/$defs/shaOrUnknown"}, | ||
| "head_sha": {"$ref": "#/$defs/shaOrUnknown"} | ||
| }, | ||
| "additionalProperties": false | ||
| }, | ||
| "patch": { | ||
| "type": "object", | ||
| "required": [ | ||
| "relation", | ||
| "mechanism", | ||
| "source_patch_id", | ||
| "target_patch_id", | ||
| "source_head_sha", | ||
| "target_base_sha", | ||
| "target_head_sha" | ||
| ], | ||
| "properties": { | ||
| "relation": {"enum": ["exact", "semantic-adaptation", "conflicted", "UNKNOWN"]}, | ||
| "mechanism": {"enum": ["git-patch-id-stable-v1", "UNKNOWN"]}, | ||
| "source_patch_id": {"$ref": "#/$defs/patchIdOrUnknown"}, | ||
| "target_patch_id": {"$ref": "#/$defs/patchIdOrUnknown"}, | ||
| "source_head_sha": {"$ref": "#/$defs/shaOrUnknown"}, | ||
| "target_base_sha": {"$ref": "#/$defs/shaOrUnknown"}, | ||
| "target_head_sha": {"$ref": "#/$defs/shaOrUnknown"} | ||
| }, | ||
| "additionalProperties": false | ||
| }, | ||
| "targetOnlyDelta": { | ||
| "type": "object", | ||
| "required": ["files", "hunks", "behavior_change", "rationale"], | ||
| "properties": { | ||
| "files": {"$ref": "#/$defs/stringListOrUnknown"}, | ||
| "hunks": {"$ref": "#/$defs/stringListOrUnknown"}, | ||
| "behavior_change": { | ||
| "oneOf": [ | ||
| {"type": "boolean"}, | ||
| {"const": "UNKNOWN"} | ||
| ] | ||
| }, | ||
| "rationale": {"$ref": "#/$defs/nonemptyStringOrUnknown"} | ||
| }, | ||
| "additionalProperties": false | ||
| }, | ||
| "reusedEvidence": { | ||
| "type": "object", | ||
| "required": ["kind", "source_id", "head_sha"], | ||
| "properties": { | ||
| "kind": {"enum": ["review", "check", "UNKNOWN"]}, | ||
| "source_id": {"$ref": "#/$defs/nonemptyStringOrUnknown"}, | ||
| "head_sha": {"$ref": "#/$defs/shaOrUnknown"} | ||
| }, | ||
| "additionalProperties": false | ||
| }, | ||
| "targetRequirements": { | ||
| "type": "object", | ||
| "required": [ | ||
| "policy_source", | ||
| "branch_protection", | ||
| "current_head_ci", | ||
| "current_head_review", | ||
| "checks" | ||
| ], | ||
| "properties": { | ||
| "policy_source": {"$ref": "#/$defs/nonemptyStringOrUnknown"}, | ||
| "branch_protection": {"enum": ["retain", "not-configured", "UNKNOWN"]}, | ||
| "current_head_ci": {"enum": ["required", "not-required", "UNKNOWN"]}, | ||
| "current_head_review": {"enum": ["required", "not-required", "UNKNOWN"]}, | ||
| "checks": {"$ref": "#/$defs/stringListOrUnknown"} | ||
| }, | ||
| "additionalProperties": false | ||
| }, | ||
| "reviewGeneratedChange": { | ||
| "type": "object", | ||
| "required": ["id", "files", "hunks", "behavior_change", "rationale"], | ||
| "properties": { | ||
| "id": {"$ref": "#/$defs/nonemptyStringOrUnknown"}, | ||
| "files": {"$ref": "#/$defs/stringListOrUnknown"}, | ||
| "hunks": {"$ref": "#/$defs/stringListOrUnknown"}, | ||
| "behavior_change": { | ||
| "oneOf": [ | ||
| {"type": "boolean"}, | ||
| {"const": "UNKNOWN"} | ||
| ] | ||
| }, | ||
| "rationale": {"$ref": "#/$defs/nonemptyStringOrUnknown"} | ||
| }, | ||
| "additionalProperties": false | ||
| }, | ||
| "forwardPortDisposition": { | ||
| "type": "object", | ||
| "required": ["change_id", "status", "rationale", "url"], | ||
| "properties": { | ||
| "change_id": {"$ref": "#/$defs/nonemptyStringOrUnknown"}, | ||
| "status": {"enum": ["applied", "tracked", "not-applicable", "UNKNOWN"]}, | ||
| "rationale": {"$ref": "#/$defs/nonemptyStringOrUnknown"}, | ||
| "url": {"$ref": "#/$defs/durableUrlOrUnknown"} | ||
| }, | ||
| "additionalProperties": false | ||
| } | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| #!/usr/bin/env ruby | ||
| # frozen_string_literal: true | ||
|
|
||
| require "json" | ||
| require_relative "../lib/verified_backport" | ||
|
|
||
| def read_utf8_input(source) | ||
| input = yield | ||
| raise EncodingError, "#{source} contains invalid UTF-8" unless input.is_a?(String) && input.valid_encoding? | ||
|
|
||
| input | ||
| end | ||
|
|
||
| begin | ||
| input = case ARGV.length | ||
| when 0 | ||
| begin | ||
| read_utf8_input("standard input") { $stdin.read } | ||
| rescue EncodingError, SystemCallError => e | ||
| warn "verified-backport-classify: unable to read input from standard input: #{e.class}: #{e.message}" | ||
| exit 66 | ||
| end | ||
| when 1 | ||
| begin | ||
| read_utf8_input("input file #{ARGV.fetch(0)}") do | ||
| File.read(ARGV.fetch(0), encoding: "UTF-8") | ||
| end | ||
| rescue EncodingError, SystemCallError => e | ||
| warn "verified-backport-classify: unable to read input file #{ARGV.fetch(0)}: #{e.class}: #{e.message}" | ||
| exit 66 | ||
| end | ||
|
justin808 marked this conversation as resolved.
|
||
| else | ||
| warn "usage: verified-backport-classify [input.json]" | ||
| exit 64 | ||
| end | ||
|
|
||
| evidence = JSON.parse(input.to_s) | ||
| puts JSON.generate(VerifiedBackport.classify(evidence)) | ||
| rescue JSON::ParserError, EncodingError | ||
| puts JSON.generate(VerifiedBackport.classify(nil)) | ||
| end | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unanchored regex pattern lets multi-line strings pass URL validation.
"pattern": "^https://[^\\s]+$"uses^/$, which in Ruby (the enginejson_schemercompiles this against) match line boundaries, not string boundaries — unlike\A/\z. A value such as"https://example.com\nanything-at-all"will validate successfully against this pattern in Ruby, even though it's clearly not a single clean durable URL.This matters here because, unlike the SHA fields (which are all re-validated with a properly-anchored
\A...\zregex inverified_backport.rb'svalid_sha?, or compared via==to an already-validated value),reviewEvidence.urlandcheckEvidence.urlare never re-checked anywhere inverified_backport.rb— onlyforwardPortDisposition.urlgets the stricterdurable_url?check (\Ahttps://[^\s]+\z). So asource_evidence.reviews[].url/checks[].urlvalue with injected trailing content after a newline would pass schema validation with no secondary guard.Same issue applies to the
shapattern on line 58/67 andrepositorypattern on line 98, though those are all independently re-validated with anchored Ruby regexes (valid_sha?,valid_source?) elsewhere, so they don't have the same exposure.Suggested fix: anchor with
\\A/\\z(or\\A...\\zequivalents that Ruby's regex engine treats as true string boundaries) in the schema pattern, e.g."^https://[^\\s]+$"→"\\Ahttps://[^\\s]+\\z".