Record App Security finding counts in command analytics

[jplhomer]

WHY are these changes introduced?

Fixes https://github.com/shop/issues-develop/issues/23969 (App Security submission telemetry).

Monorail app_cli3_command/1.30 (Shopify/monorail#24622) added num_security_findings and num_security_findings_resolved, but the CLI still sends 1.28 and none of the app security commands record any counts. Separately, INSTRUCTIONS.md tells agents "Telemetry is disabled for this workflow", which isn't true: the standard command event is sent for every run.

WHAT is this pull request doing?

• Bumps MONORAIL_COMMAND_TOPIC to app_cli3_command/1.30 and adds both fields as optional numbers. I checked the 1.30 schema: it contains every field the CLI sends except cmd_app_validate_*, which 1.28 didn't have either, so nothing regresses.
• Adds both fields to the app metadata container. They don't match any PickByPrefix.
• Sets num_security_findings (through a new injected recordMetadata dependency, following the existing DI pattern):
  • check: the number of deterministic issues, the same number as "N security issues found".
  • record: the number of findings in the recorded agent document, the same number as "Recorded N findings". A rejected document records nothing.
  • review: the number of active findings across every combined check. That excludes suppressed findings and findings superseded under prefer-agent, and it ignores --check-id so the number doesn't depend on the filter. Left unset when neither results file exists. review is the only command that sees the combined view, so this is the best "issues remaining" signal.
• Leaves num_security_findings_resolved unset. Stored findings have no stable identity (only file/line/message), so comparing with the previous run would count moved code as resolved + new. The trend can be computed in the warehouse from successive per-app counts until findings get an identity.
• Rewords the INSTRUCTIONS.md telemetry line: the CLI sends standard usage telemetry including the number of findings, doesn't send source code, finding details, or artifacts, and users can opt out with SHOPIFY_CLI_NO_ANALYTICS=1 (per analyticsDisabled() in cli-kit). Regenerated checks/embedded.ts.

Not in scope: api_key / organization_id attribution. That fits better on top of #8736 / #8737, which move the security commands onto localAppContext with the selected --path / --config.

Expected conflicts with #8736 / #8737: security-check.ts and security-record.ts signatures, and INSTRUCTIONS.md (a different line). Each recordMetadata call is one line, so the rebase is trivial either way. embedded.ts needs regenerating.

No changeset: the commands are still hidden.

How to manually test your changes?

With SHOPIFY_CLI_ENV=development the CLI skips sending analytics and logs the payload instead, so nothing reaches Monorail:

1. In an app, run SHOPIFY_CLI_ENV=development shopify app security check --path <app> --verbose. In the Skipping command analytics, payload: debug line, look for num_security_findings matching the reported issue count.
2. Pipe a findings document to shopify app security record --path <app> --verbose, then run shopify app security review --path <app> --verbose, and check num_security_findings in each payload. The topic constant is app_cli3_command/1.30.

Checklist

• I've considered possible cross-platform impacts (Mac, Linux, Windows)
• I've considered possible documentation changes
• I've considered analytics changes to measure impact
• The change is user-facing — hidden commands, no changeset

Warehouse follow-up (separate repos): the new columns still need selecting in base__monorail_app_cli3_command.sql and cli3_commands_v1.sql.

---

PR authored by Qlaw

[Copilot]

Copilot review overview

🟢 Approval recommended

Finding-count semantics are correctly implemented and tested, and the generated instructions are synchronized.

Review effort: Balanced
Findings: None

What changed in this PR

Adds App Security finding counts to standard command analytics while clarifying telemetry behavior.

Changes:

• Upgrades the Monorail schema and exposes finding-count metadata.
• Records appropriate counts for security check, record, and review commands.
• Updates tests and regenerated agent instructions.

File	Description
packages/​cli-kit/​src/​public/​node/​monorail.ts	Updates topic and analytics fields.
packages/​app/​src/​cli/​metadata.ts	Exposes security metadata fields.
packages/​app/​src/​cli/​services/​app-security-metadata.ts	Adds metadata recording helper.
packages/​app/​src/​cli/​services/​security-check.ts	Records deterministic finding count.
packages/​app/​src/​cli/​services/​security-check.test.ts	Tests check telemetry counts.
packages/​app/​src/​cli/​services/​security-record.ts	Records accepted agent findings.
packages/​app/​src/​cli/​services/​security-record.test.ts	Tests record telemetry behavior.
packages/​app/​src/​cli/​services/​security-review.ts	Records all active combined findings.
packages/​app/​src/​cli/​services/​security-review.test.ts	Tests filtering and missing results.
packages/​app/​src/​cli/​services/​app-security-engine/​INSTRUCTIONS.md	Clarifies telemetry and opt-out behavior.
packages/​app/​src/​cli/​services/​app-security-engine/​checks/​embedded.ts	Regenerates embedded instructions.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Differences in type declarations

We detected differences in the type declarations generated by Typescript for this branch compared to the baseline ('main' branch). Please, review them to ensure they are backward-compatible. Here are some important things to keep in mind:

• Some seemingly private modules might be re-exported through public modules.
• If the branch is behind main you might see odd diffs, rebase main into this branch.

New type declarations

We found no new type declarations in this PR

Existing type declarations

packages/cli-kit/dist/public/node/monorail.d.ts

@@ -2,7 +2,7 @@ import { JsonMap } from '../../private/common/json.js';
 import { DeepRequired } from '../common/ts/deep-required.js';
 export { DeepRequired };
 type Optional<T> = T | null;
-export declare const MONORAIL_COMMAND_TOPIC = "app_cli3_command/1.28";
+export declare const MONORAIL_COMMAND_TOPIC = "app_cli3_command/1.30";
 export interface Schemas {
     [MONORAIL_COMMAND_TOPIC]: {
         sensitive: {
@@ -77,6 +77,8 @@ export interface Schemas {
             cmd_app_validate_valid?: Optional<boolean>;
             cmd_app_validate_issue_count?: Optional<number>;
             cmd_app_validate_file_count?: Optional<number>;
+            num_security_findings?: Optional<number>;
+            num_security_findings_resolved?: Optional<number>;
             cmd_dev_tunnel_type?: Optional<string>;
             cmd_dev_tunnel_custom_hash?: Optional<string>;
             cmd_dev_urls_updated?: Optional<boolean>;