App Security: let agents choose, check and rerun a scope by jek · Pull Request #8740 · Shopify/cli

jek · 2026-10-02T12:59:49Z

WHY are these changes introduced?

With --include-dir, --exclude and --no-git-ignore, an agent needs to settle on a scope, verify it, rerun it the same way, and record which scope its findings cover.

WHAT is this pull request doing?

check --list-files prints the gathered paths (or {"files": [...]} with --json) without scanning.
Generated commands render --path, the selection and the typed scope flags in a fixed order, relative to the working directory, which the instructions now name.
Bare instructions tell the agent to choose the scope with --list-files first.
check records the scope and scan directories in coverage; record requires the agent's scope; review shows both and notes when they differ.
Adds the layout catalogue tests: 27 real-repository layouts asserting selection, gathered paths and rerun commands.

How to manually test your changes?

shopify app security instructions
shopify app security check --include-dir ../backend --list-files
shopify app security check --include-dir ../backend
shopify app security review

Checklist

I've considered possible cross-platform impacts (Mac, Linux, Windows)
I've considered possible documentation changes
I've considered analytics changes to measure impact
The change is user-facing — I've identified the correct bump type (patch for bug fixes · minor for new features · major for breaking changes) and added a changeset with pnpm changeset add

jek · 2026-10-02T13:05:07Z

/snapit

github-actions · 2026-10-02T13:08:15Z

🫰✨ Thanks @jek! Your snapshot has been published to npm.

Built from 776d9cac4eb25c8fbb31658c909b65815bfe959b. Workflow run.

Test the snapshot by installing your package globally:

pnpm i -g --@shopify:registry=https://registry.npmjs.org @shopify/cli@0.0.0-snapshot-20261002130601

Caution

After installing, validate the version by running shopify version in your terminal.
If the versions don't match, you might have multiple global instances installed.
Use which shopify to find out which one you are running and uninstall it.

github-actions · 2026-10-02T20:24:23Z

Differences in type declarations

We detected differences in the type declarations generated by Typescript for this branch compared to the baseline ('main' branch). Please, review them to ensure they are backward-compatible. Here are some important things to keep in mind:

Some seemingly private modules might be re-exported through public modules.
If the branch is behind main you might see odd diffs, rebase main into this branch.

New type declarations

We found no new type declarations in this PR

Existing type declarations

packages/cli-kit/dist/public/node/api/app-management.d.ts

@@ -7,7 +7,6 @@ export declare const appManagementAppLogsUrl: (organizationId: string, cursor?:
     status?: string;
     source?: string;
 }) => Promise<string>;
-export declare const appManagementChannelSpecExportUrl: (organizationId: string, appId: string) => Promise<string>;
 export interface RequestOptions {
     requestMode: RequestModeInput;
 }

Add check --list-files, render rerun commands from the selection and the typed scope flags, and record the scope on both sides so review can flag a mismatch. The instructions name the working directory and have the agent settle the scope before scanning. Cover the layout catalogue.

Run the real command with --without-app-config and --exclude, and check the results key, the selection, the recorded scope and the unresolved config checks. No other test scans with no app configuration file.

jplhomer

Agentic check ran successfully 👍

dmerand

Small change request to maintain backwards compatibility. Should be quick and/or we can discuss.

dmerand · 2026-10-05T16:40:01Z

  ...documentBase,
  source: zod.literal('agent'),
  engine: zod.object({name: zod.literal(ENGINE_NAME), version: zod.string()}),
+  scope: createScopeSchema(),


This comment is the change request:

scope is required while schema_version stays 1, so stored results from an earlier CLI can no longer be read.

I verified this with the compiled CLI: agent-findings.json recorded before this change has no scope, and review now exits 1 with "The App Security results could not be loaded because a results file is invalid" and the detail scope: Required. The same applies to deterministic findings without the new coverage.scope and coverage.scan_directories. The record input also stays at version 1 while requiring scope.

If old results should keep loading, please bump the version or add a legacy representation (for example an unknown-scope marker with guidance to run check again), and make the error explain the format change. Please don't invent an empty scope for historical results — that would misstate what was scanned.

We're intentionally not keeping compatibility here.

EA wrote a flat .shopify/app-security/*.json layout that this stack no longer reads; the keyed / layout and the scope fields ship in the same merge, so no released CLI writes a keyed results file without scope.

A scope-less file can only come from an intermediate stack build (#8737–#8739 alone), which won't ship. Version 1 is effectively the first version of the keyed layout. Pre-GA, we don't carry EA formats forward.

Great, makes sense. I'll adjust to approved + leave the rest to your discretion.

review printed a check command without the scope, so following it scanned less (or more) than the results it summarised.

Every other free-form value in the results files is redacted, but the scope's --include-dir and --exclude values and the scan directories were stored as typed. Both results files redact them the same way, so they still compare equal.

"Nothing is recorded" read as "nothing is written", but selecting the app can create .shopify/project.json as every app command does.

github-actions · 2026-10-05T17:43:34Z

⚠️ Potential Breaking Changes Detected

This PR contains changes that may break the existing contract.

@shopify/dev_experience — this PR contains breaking changes that require coordination for the next major release.

🏳️ Removed Flags

The following flags were removed from existing commands:

Command	Flag
`app:security:check`	`--ignore`

jek added this pull request to stack #8741 October 2, 2026 12:59

github-actions Bot added the no-changelog This PR doesn't include a changeset entry. Is an internal only change not relevant to end users. label Oct 2, 2026

jek changed the title ~~Let agents choose, check and rerun an App Security scope~~ App Security: let agents choose, check and rerun a scope Oct 2, 2026

jek force-pushed the app-security/agent-workflow branch from 776d9ca to 2f30f7f Compare October 2, 2026 13:35

jek force-pushed the app-security/agent-workflow branch 2 times, most recently from 26686db to c23b4d8 Compare October 2, 2026 14:15

jek marked this pull request as ready for review October 2, 2026 18:36

jek requested review from a team as code owners October 2, 2026 18:36

jek requested a review from jplhomer October 2, 2026 18:38

jek force-pushed the app-security/agent-workflow branch from c23b4d8 to e57a89c Compare October 2, 2026 20:20

dmerand mentioned this pull request Oct 2, 2026

App Security: resolve the app selection once #8736

Merged

4 tasks

jek force-pushed the app-security/agent-workflow branch from e57a89c to b05efb6 Compare October 2, 2026 21:07

jek added 2 commits October 5, 2026 06:43

Test an App Security check without app configuration end to end

181ddef

Run the real command with --without-app-config and --exclude, and check the results key, the selection, the recorded scope and the unresolved config checks. No other test scans with no app configuration file.

jek force-pushed the app-security/agent-workflow branch from b05efb6 to 181ddef Compare October 5, 2026 13:46

dmerand mentioned this pull request Oct 5, 2026

App Security: store results per results key #8737

Merged

4 tasks

jplhomer approved these changes Oct 5, 2026

View reviewed changes

dmerand requested changes Oct 5, 2026

View reviewed changes

dmerand approved these changes Oct 5, 2026

View reviewed changes

jek added 3 commits October 5, 2026 10:31

Repeat the latest scan's scope in review's check command

cbaaec0

review printed a check command without the scope, so following it scanned less (or more) than the results it summarised.

Redact secrets in the recorded App Security scope

eeb396d

Every other free-form value in the results files is redacted, but the scope's --include-dir and --exclude values and the scan directories were stored as typed. Both results files redact them the same way, so they still compare equal.

Say that --list-files writes no results

a3f10b6

"Nothing is recorded" read as "nothing is written", but selecting the app can create .shopify/project.json as every app command does.

jek requested a review from dmerand October 5, 2026 17:46

dmerand approved these changes Oct 5, 2026

View reviewed changes

jek added this pull request to the merge queue Oct 5, 2026

Merged via the queue into main with commit 6ee8b14 Oct 5, 2026
29 of 30 checks passed

jek deleted the app-security/agent-workflow branch October 5, 2026 18:17

Navigation Menu

App Security: let agents choose, check and rerun a scope - #8740

App Security: let agents choose, check and rerun a scope#8740
jek merged 5 commits into
app-security/include-dirfrom
app-security/agent-workflow

jek commented Oct 2, 2026 •

edited

Loading

Uh oh!

jek commented Oct 2, 2026

Uh oh!

github-actions Bot commented Oct 2, 2026

Uh oh!

github-actions Bot commented Oct 2, 2026

Uh oh!

jplhomer left a comment

Uh oh!

dmerand left a comment

Uh oh!

dmerand Oct 5, 2026

Uh oh!

jek Oct 5, 2026

Uh oh!

dmerand Oct 5, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Oct 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jek commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

WHY are these changes introduced?

WHAT is this pull request doing?

How to manually test your changes?

Checklist

Uh oh!

jek commented Oct 2, 2026

Uh oh!

github-actions Bot commented Oct 2, 2026

Uh oh!

github-actions Bot commented Oct 2, 2026

Differences in type declarations

New type declarations

Existing type declarations

Uh oh!

jplhomer left a comment

Choose a reason for hiding this comment

Uh oh!

dmerand left a comment

Choose a reason for hiding this comment

Uh oh!

dmerand Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

jek Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

dmerand Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Oct 5, 2026

⚠️ Potential Breaking Changes Detected

🏳️ Removed Flags

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jek commented Oct 2, 2026 •

edited

Loading