Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions products/signals/backend/test/test_scout_harness_lazy_seed.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
from __future__ import annotations

import re
import textwrap
from datetime import UTC, datetime, timedelta
from pathlib import Path
Expand Down Expand Up @@ -36,6 +37,8 @@
)
from products.skills.backend.models.skills import LLMSkill, LLMSkillFile

from ee.billing.billing_types import USAGE_TYPE_VALUES

_SCHEMA_JSON = '{"type": "object", "properties": {"verdict": {"type": "string"}}}'


Expand Down Expand Up @@ -812,6 +815,23 @@ def test_mcp_tool_calls_schema_accepts_one_record_of_each_kind(self) -> None:
_validate_records([StructuredOutputRecord(payload=rejected)], schema)


class TestShippedBillingScoutVocabulary:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Org billing figures reach project members without billing access, enforced only by skill prose

The shipped signals-scout-billing skill (in products/signals/skills/signals-scout-billing/) reads the organization's bill under an acting user with billing:read, then publishes dollar deltas, spending-limit proximity, and trial-end warnings into inbox reports, the scratchpad, and Slack — surfaces gated only by team membership (ScoutCanonicalTeamAccessPermission, products/signals/backend/presentation/scout_permissions.py:13), never by billing access. The skill's own text admits it: "anyone who can read the inbox reads your billing numbers whether or not the billing pages would let them in" (SKILL.md:257). The sole control is LLM instruction-following ("carry only the dollar delta", SKILL.md:258; "memory holds volumes, dates, and verdicts, never billing detail", SKILL.md:240), and the PR lists "gate report visibility by billing access in code" as an undone follow-up — so an org admin's billing read is summarized into team- and Slack-visible facts for members the billing pages deny.

How:

  1. An org admin enables the scout (or is its acting user), so the billing tools resolve.
  2. Any run files a report such as "adds ~$180 to this period" plus limit/trial warnings.
  3. A project member with no billing access reads the inbox report or the Slack delivery — no code check exists.
  4. Prose is the only filter, so a drifting or injected run (e.g. a customer-controlled project name pushing it to restate totals) has nothing in code to stop it.

Fix: gate this scout's report/scratchpad reads (or its emission content) on billing access in code, as the PR's own follow-up already plans.


React with 👍 if useful or 👎 if not

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid, and accepted for now rather than fixed here. The scout registers disabled on every project, so a person with billing access has to turn it on before any report exists. The code gate on report and memory reads is the follow-up before it is enabled by default. 72628f5 also removes the two calls that could read other projects' usage.

def test_every_billing_usage_type_has_a_product_key_row(self) -> None:
# The scout prices and routes a meter only through this table, so a usage type billing adds
# without a row here is a meter the scout reads but can neither price nor hand off.
(skill,) = [s for s in discover_canonical_skills() if s.name == "signals-scout-billing"]
(reference,) = [f.content for f in skill.files if f.path == "references/usage-types.md"]
vocabulary = reference.split("## Three vocabularies for one meter", 1)[1].split("\n## ", 1)[0]
mapped = {
usage_type
for line in vocabulary.splitlines()
if line.startswith("| `")
for usage_type in re.findall(r"`([a-z0-9_]+)`", line.split("|")[1])
}

assert set(USAGE_TYPE_VALUES) - mapped == set()


class TestDeprecationFrontmatter:
"""`scout-status` / `scout-deprecation`: the marker a retirement PR adds to a SKILL.md."""

Expand Down
13 changes: 12 additions & 1 deletion products/signals/skills/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,17 @@ A scout is a skill that holds a `SignalScoutConfig`. The harness globs `signals-
Its discriminator is a statically-verifiable rule violation in a skill that is fresh or load-bearing — the unchanged long tail, subjective style nits, and canonical seeded scout rows (`category: "scout"`) are noise. Treats skill bodies as untrusted data under test, never instructions.
- `signals-scout-customer-analytics/` — account-health watcher for the Customer analytics (Accounts) product, where each `system.accounts` row is a customer organization keyed to its analytics by `external_id` (the group key). Curates a watchlist of commercially-staked accounts (assigned CSM / AE / owner, or a CRM link) and scores each account's product engagement — weekly volume / WAU / key-feature usage — against its own trailing baseline, watching for churn-risk shapes (an engagement cliff, dormancy onset, single-threaded champion departure) and the positive inverse (an expansion signal worth an upsell). Its discriminator is a per-account engagement regression while the fleet holds, weighted by commercial ownership: one staked account sliding is signal, the whole fleet moving together is a capture/aggregate problem for another scout. The linchpin is the account→group join — it verifies that `external_id` actually matches a live group key before trusting any per-account number (a seeded/CRM-sourced roster that doesn't join is a config gap, not a finding). Fills the seam neither commercial neighbor covers — `product-analytics` scores aggregate user-grain flows and `revenue-analytics` watches the lagging revenue/MRR signal; neither scores an individual account's engagement trajectory. Acquisition is the web-analytics scout's territory.
- `signals-scout-customer-analytics-billing-and-usage/` — per-account product-mix watcher: one product's usage or forecasted MRR moving sharply in either direction against that account+product's own same-weekday trailing 4-week baseline **while the account total holds** — the masked shift account-level monitoring misses ("flags down 40%, total flat — replay growth is masking it"). Reads per-account per-product usage and MRR from the billing warehouse sources it discovers at Orient, weights severity by % change × the product's share of the account's MRR (an unexplained bill-inflating spike ranks with drops — surprise invoices churn accounts too), and sweeps account notes, channel summaries, and synced comms for a planned-change explanation before filing. Its discriminator is per-product divergence under a flat account total on a commercially staked account. Fills the seam its two commercial neighbors leave open — `customer-analytics` scores the account's aggregate engagement (a flat total looks healthy to it) and `revenue-analytics` watches aggregate revenue and upstream sync health; fleet-wide product moves are `product-analytics` / `health-checks` territory. On the **report channel** (`emit_report` / `edit_report`): one report per account+product shift, edited while the shift persists. **Withheld fleet-wide** via the `signals-scout` flag's `default_team_config.withheld_skills` and released only on team 2 (its `team_configs` override) — the billing views and the `postgres.posthog_user` reviewer mapping it depends on exist only there.
- `signals-scout-billing/` — watcher for the organization's **own PostHog bill**, read through the customer-facing billing tools (`billing-usage-get`, `billing-spend-get`, `billing-overview-get`) rather than any internal billing source.
Two lanes per run: a **step** lane that scores each usage type's daily volume on this project (`team_ids` pinned) against its own same-weekday baseline, and an org-level **trajectory** lane that reads projected period spend, spending limits, and trial end from the overview.
Its discriminator is machine-shaped × material × unexplained movement, with materiality priced as `price(with) − price(without)` across the tier schedule, never as a percentage or `delta × rate`: a big move inside the free tier costs nothing, and tiers reset each period, so a spend comparison across a period boundary manufactures spikes.
Several meters stepping together are not dismissed as traffic: one events-versus-distinct-people read separates growth (people up with usage) from an instrumentation change (usage up, people flat), which is one report naming every meter that moved.
It owns that the bill moved and hands root cause to the owning product scout through the usage-type → product key → scout table in bundled `references/usage-types.md`, which maps every usage type in `ee/billing/billing_types.py` (a test holds the two together).
A missing billing tool is a close-out and a denial is never evidence that usage fell: a usage-only caller runs the step lane without dollar figures, and a caller with no billing access scores this project's event volume as a stand-in.
Either denial files one report telling the owner the scout cannot read billing data and how to fix it, because it reads billing as the person who turned it on.
Scratchpad memory holds volumes, dates, and verdicts only, because project members without billing access can read it; a report states only the dollar delta of the one move, never totals, tier prices, or plan detail.
Its default pre-check starts a scheduled run about weekly, or sooner when yesterday's events doubled against the same weekday a week before, and returns no rows on a project with no events in 8 days.
The trajectory lane is organization-wide while reports are team-scoped, so enable it on one project per organization.
Distinct from `revenue-analytics` (the customer's own revenue) and `customer-analytics-billing-and-usage` (other organizations' usage from internal warehouse views).
- `signals-scout-mcp-tool-calls/` — tool-quality watcher for PostHog MCP usage. Reads `$mcp_tool_call` telemetry for tools that need improvement: high, broad-reach failure rates, per-session retry/hammering that betrays a confusing schema, slow or context-bloating responses, and undiagnosable failures (an instrumentation gap). Its discriminator is rate/struggle weighted by volume and reach, concentrated in a consistent shape — not raw counts. Coverage-aware: it first probes which enrichment fields the project captures (they split by regime — PostHog's own hono server vs external SDK servers) and picks lenses to match, resting detection only on always-present fields (error flag, duration, tool name, session). On the **report channel** (`emit_report` / `edit_report`): files one report per problem category (`$mcp_tool_category`, the owning product team) listing that category's problem tools each with a fix hypothesis, editing the live category report while the category still has problem tools; falls back to one report per tool where category coverage is absent (external-SDK regime); bundles `references/queries.md`, a HogQL cookbook validated against real telemetry. The one scout on the **structured-output channel**: `scout-structured-output-schema` in its frontmatter ships `references/structured-output.schema.json`, so every run records a per-category roll-up and per-tool session-share rows as `$scout_structured_output` events — a chartable series for numbers that exist nowhere else (struggle share, problem-tool counts, the regime, and what the scout decided per category), with `references/metrics-dashboard.md` as the consumption recipe.
- `signals-scout-tasks/` — delivery + demand watcher for the Tasks product (the agent work items a project runs), read from the `system.tasks` / `system.task_runs` system tables. Two lenses in one body: **delivery health** every run (runs failing, clustered by repository and by `error_message` prefix) and a ~weekly gated **demand** pass over human-authored tasks (recurring asks that point at a product gap). Its discriminator is failure concentration × spread — a cluster's failure rate over meaningful volume, with runs ÷ distinct tasks separating a _systemic_ defect in a shared path (ratio ≈ 1 across many tasks) from a _retry storm_ (ratio ≫ 1 over one or two stuck tasks), which is the single biggest false positive on this surface. Two exclusions are load-bearing: it always drops `origin_product = 'signals_scout'` (the fleet's own run containers, which the `internal` flag does _not_ filter and which can outnumber every real origin), and the demand lens reads human origins only, so the inbox's own throughput can never register as user demand. On the **report channel** (`emit_report` / `edit_report`): one report per cluster (a repository or a failure class), with the demand lens held to a higher bar — its default outcome is compounding scratchpad memory, and it files only once a theme crosses into something actionable. Bundles `references/queries.md`, a cookbook validated against real task data (it also tells the scout to probe `stage` before resting a lens on it). Task text is treated as untrusted prompt material, never instructions.
- `signals-scout-conversations/` — support-delivery health watcher for the Conversations (support inbox) product. Reads the `$conversation_*` ticket-lifecycle events for operational regressions: SLA breach-rate steps on team replies (`sla_active` / `sla_breached`), first-response latency blowouts (ticket-created → first team reply), backlog inflow-vs-resolution imbalance, and channel / assignment / priority concentration. Its discriminator is a _rate against a volume-stable denominator, per operational dimension, stepping away from its own trailing baseline while ticket volume holds_ — a breach share / response latency / inflow-minus-resolution delta moving on steady volume is signal, a raw count that tracks inbound volume is baseline — always gated by a minimum-volume guard. On the **report channel** (`emit_report` / `edit_report`): files each dated, dimension-named regression as a 1:1 inbox report, editing the live report while the regression persists. It is the operational complement to the per-ticket emission path: the emission pipeline (`source_product="conversations"`) already reads each ticket thread and fires a per-ticket _product-feedback_ signal (bugs / feature requests / usability), so this scout never re-surfaces individual ticket content — it owns the aggregate throughput / SLA / backlog / routing shape the one-ticket-at-a-time emitter can't see, and (reading analytics events) it works whether or not that emission source is enabled.
Expand Down Expand Up @@ -140,7 +151,7 @@ The legacy signal channel (weak `emit_signal` findings for the pipeline to clust
The user-facing skills (`authoring-scouts`, `exploring-scouts`) no longer teach it; new scouts that author reports list the report tools.

The specialists each carry their own domain discriminator + investigation patterns.
Most are a single self-contained `SKILL.md`; a few bundle surface-specific references read on demand — `signals-scout-anomaly-detection` (`legacy-detection.md`, `anomaly-methods.md`, `watchlist-and-memory.md`, `report-contract.md`, `source-liveness.md`), `signals-scout-ai-observability` (`lenses.md`), `signals-scout-feature-flags` (`stale-flags.md`), `signals-scout-apm` (`performance-issues.md`), `signals-scout-surveys` (`response-querying.md`), `signals-scout-web-vitals` (`remediation.md`, `onset-correlation.md`), `signals-scout-mcp-tool-calls` (`queries.md`, `metrics-dashboard.md`, `structured-output.schema.json`), `signals-scout-tasks` (`queries.md`), `signals-scout-pr-follow-up` (`sources.md`, `deploy-ladder.md`, `probes.md`), `signals-scout-product-analytics` (`funnel-maturity.md`), and `signals-scout-unused-features` (`queries.md`); `ls signals-scout-*/references/` is the current list. Treat the generalist as the reference shape. Note that a scout can only read its own bundled files at runtime (each team's `LLMSkill` row carries just that skill's files) — which is why a scout that needs the dedupe/memory conventions in depth bundles its own copy rather than pointing at the generalist's. The report-channel contract is the exception to that copy-it-locally rule: it rides in the harness prompt, not a file, so report scouts share one definition without each bundling a copy.
Most are a single self-contained `SKILL.md`; a few bundle surface-specific references read on demand — `signals-scout-anomaly-detection` (`legacy-detection.md`, `anomaly-methods.md`, `watchlist-and-memory.md`, `report-contract.md`, `source-liveness.md`), `signals-scout-ai-observability` (`lenses.md`), `signals-scout-feature-flags` (`stale-flags.md`), `signals-scout-apm` (`performance-issues.md`), `signals-scout-surveys` (`response-querying.md`), `signals-scout-web-vitals` (`remediation.md`, `onset-correlation.md`), `signals-scout-mcp-tool-calls` (`queries.md`, `metrics-dashboard.md`, `structured-output.schema.json`), `signals-scout-tasks` (`queries.md`), `signals-scout-pr-follow-up` (`sources.md`, `deploy-ladder.md`, `probes.md`), `signals-scout-product-analytics` (`funnel-maturity.md`), `signals-scout-billing` (`usage-types.md`), and `signals-scout-unused-features` (`queries.md`); `ls signals-scout-*/references/` is the current list. Treat the generalist as the reference shape. Note that a scout can only read its own bundled files at runtime (each team's `LLMSkill` row carries just that skill's files) — which is why a scout that needs the dedupe/memory conventions in depth bundles its own copy rather than pointing at the generalist's. The report-channel contract is the exception to that copy-it-locally rule: it rides in the harness prompt, not a file, so report scouts share one definition without each bundling a copy.

## When editing skills in this directory

Expand Down
Loading
Loading