Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
5ab7eeb
feat(signals): add canonical billing scout
andrewm4894 Aug 20, 2026
f1a545a
fix(signals): correct billing scout field semantics and scoring band
andrewm4894 Aug 20, 2026
50a1cbe
fix(signals): align billing scout with live billing tool responses
andrewm4894 Aug 20, 2026
f4750bb
fix(signals): make billing scout gates self-consistent and callable
andrewm4894 Aug 20, 2026
e45268e
fix(signals): let the billing scout degrade gracefully on missing bil…
andrewm4894 Aug 20, 2026
c3e3eb0
fix(signals): address billing scout review on zero days, add-ons, and…
andrewm4894 Aug 20, 2026
68cd2fe
fix(signals): tighten billing scout lag detection, families, and peri…
andrewm4894 Aug 20, 2026
ae34042
chore(signals): reformat the billing scout meter-family table
andrewm4894 Aug 20, 2026
b4b3252
fix(signals): correct billing scout cursor, limit, and pricing edge c…
andrewm4894 Aug 20, 2026
bca6346
Merge branch 'master' into posthog/signals-scout-billing
andrewm4894 Aug 29, 2026
fcc51f0
fix(signals): score gap days and anchor billing pricing to one baseline
andrewm4894 Aug 29, 2026
93a273b
Merge remote-tracking branch 'origin/master' into posthog/signals-sco…
andrewm4894 Nov 18, 2026
a729283
fix(signals): keep billing detail out of scout memory and tighten pri…
andrewm4894 Nov 18, 2026
ad6be3e
fix(signals): check people on lone event steps and search reports by …
andrewm4894 Nov 18, 2026
6008c80
feat(signals): run the billing scout weekly behind a pre-check
andrewm4894 Nov 18, 2026
113c960
fix(signals): keep the billing pre-check quiet on an idle project
andrewm4894 Nov 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions products/signals/skills/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ A scout is a skill that holds a `SignalScoutConfig`. The harness globs `signals-
Its discriminator is a statically-verifiable rule violation in a skill that is fresh or load-bearing — the unchanged long tail, subjective style nits, and canonical seeded scout rows (`category: "scout"`) are noise. Treats skill bodies as untrusted data under test, never instructions.
- `signals-scout-customer-analytics/` — account-health watcher for the Customer analytics (Accounts) product, where each `system.accounts` row is a customer organization keyed to its analytics by `external_id` (the group key). Curates a watchlist of commercially-staked accounts (assigned CSM / AE / owner, or a CRM link) and scores each account's product engagement — weekly volume / WAU / key-feature usage — against its own trailing baseline, watching for churn-risk shapes (an engagement cliff, dormancy onset, single-threaded champion departure) and the positive inverse (an expansion signal worth an upsell). Its discriminator is a per-account engagement regression while the fleet holds, weighted by commercial ownership: one staked account sliding is signal, the whole fleet moving together is a capture/aggregate problem for another scout. The linchpin is the account→group join — it verifies that `external_id` actually matches a live group key before trusting any per-account number (a seeded/CRM-sourced roster that doesn't join is a config gap, not a finding). Fills the seam neither commercial neighbor covers — `product-analytics` scores aggregate user-grain flows and `revenue-analytics` watches the lagging revenue/MRR signal; neither scores an individual account's engagement trajectory. Acquisition is the web-analytics scout's territory.
- `signals-scout-customer-analytics-billing-and-usage/` — per-account product-mix watcher: one product's usage or forecasted MRR moving sharply in either direction against that account+product's own same-weekday trailing 4-week baseline **while the account total holds** — the masked shift account-level monitoring misses ("flags down 40%, total flat — replay growth is masking it"). Reads per-account per-product usage and MRR from the billing warehouse sources it discovers at Orient, weights severity by % change × the product's share of the account's MRR (an unexplained bill-inflating spike ranks with drops — surprise invoices churn accounts too), and sweeps account notes, channel summaries, and synced comms for a planned-change explanation before filing. Its discriminator is per-product divergence under a flat account total on a commercially staked account. Fills the seam its two commercial neighbors leave open — `customer-analytics` scores the account's aggregate engagement (a flat total looks healthy to it) and `revenue-analytics` watches aggregate revenue and upstream sync health; fleet-wide product moves are `product-analytics` / `health-checks` territory. On the **report channel** (`emit_report` / `edit_report`): one report per account+product shift, edited while the shift persists. **Withheld fleet-wide** via the `signals-scout` flag's `default_team_config.withheld_skills` and released only on team 2 (its `team_configs` override) — the billing views and the `postgres.posthog_user` reviewer mapping it depends on exist only there.
- `signals-scout-billing/` — watcher for the organization's **own PostHog bill**, read through the customer-facing billing MCP tools (`billing-overview-get` / `billing-usage-get` / `billing-spend-get`) rather than any internal billing source. Two lanes per run: a **step** lane scoring each usage type's daily volume on **this project** against its own same-weekday trailing baseline (the overnight instrumentation loop, the debug log level left on, the runaway sync — including the instrumentation change that lifts several meters at once while the number of people using the product holds), and an org-level **trajectory** lane reading the overview's capped projected period spend, `custom_limits_usd`, and tier position (the slow ramp no daily z-score trips, and spending limits about to be crossed). Its discriminator is machine-shaped × material × unexplained movement, with materiality priced **against the tier schedule** — a big percentage on a meter still inside its free tier costs nothing, which is the surface's biggest false positive, and the free threshold comes from the tier schedule rather than `free_allocation` (a subscribed product can ship `free_allocation: 0` with its first million units free). The billing-specific trap it is built around: tiers reset each period, so spend is not proportional to usage and a cross-boundary spend comparison manufactures spikes out of arithmetic. The body states the band numerically — robust z ≥ 3.5 against four same-weekday values, a ≥50% fallback when MAD is zero, a 100-units/day volume floor applied to the baseline **and** the latest value (so a cold-start spike off a tiny baseline still scores), ±20% concentration tolerance measured over **meter families** rather than individual series (alias/subset meters like `group_analytics` and the logs retention tier would otherwise read as a second moving meter), with a co-moving step separated from traffic growth by one events-per-person read rather than dropped, and a materiality floor of the greater of $20 and 2% of the last period's invoice — so two runs reach the same verdict on the same data. Dollar impact is `price(with) − price(without)` across the tier schedule, then discount and cap, never `delta × rate`. It owns **that the bill moved** and hands root cause to the owning product scout (bundles `references/usage-types.md`: the three-vocabulary map the tools force on a caller — usage-get keys by usage type, spend-get and `custom_limits_usd` by product, `usage_key` by short name — plus free-tier and delta-pricing derivation, which of the four projection fields to quote, the `never_drop_data` caveat on limit consequences, both trial representations, and the usage-type → scout routing table); it attaches no `charts`, since billing series come from a REST tool and have no HogQL node. Ships a default pre-check (`scout-precheck-query`) that starts a scheduled run about once a week, and sooner when the project's event volume doubles against the same weekday a week before. Three access gates change what a run can conclude — tools absent (a deployment with no billing service) and permission denied are close-outs, while the usage-only member grant drops the trajectory lane and puts the step lane in **volume-only mode**, where it reports just the two shapes decisive without a dollar figure (a meter going dark, and a ≥5× step holding 3+ days) and carries the rest as memory. The step lane's cursor close-out never gates the trajectory lane, so a sparse designated project cannot suppress a spending-limit or trial-expiry warning driven by a sibling project. **Release it to one project per organization:** the step lane pins `team_ids` to its own project so it dedupes by construction, but the trajectory lane is inherently org-wide, and scouts, scratchpads, and inbox reports are all team-scoped — so two enabled projects in one org would each file the same trajectory finding into different inboxes. Distinct from both money neighbors: `revenue-analytics` watches the customer's own revenue from their customers, and `customer-analytics-billing-and-usage` watches _other_ organizations' usage from internal billing warehouse views.
- `signals-scout-mcp-tool-calls/` — tool-quality watcher for PostHog MCP usage. Reads `$mcp_tool_call` telemetry for tools that need improvement: high, broad-reach failure rates, per-session retry/hammering that betrays a confusing schema, slow or context-bloating responses, and undiagnosable failures (an instrumentation gap). Its discriminator is rate/struggle weighted by volume and reach, concentrated in a consistent shape — not raw counts. Coverage-aware: it first probes which enrichment fields the project captures (they split by regime — PostHog's own hono server vs external SDK servers) and picks lenses to match, resting detection only on always-present fields (error flag, duration, tool name, session). On the **report channel** (`emit_report` / `edit_report`): files one report per problem category (`$mcp_tool_category`, the owning product team) listing that category's problem tools each with a fix hypothesis, editing the live category report while the category still has problem tools; falls back to one report per tool where category coverage is absent (external-SDK regime); bundles `references/queries.md`, a HogQL cookbook validated against real telemetry. The one scout on the **structured-output channel**: `scout-structured-output-schema` in its frontmatter ships `references/structured-output.schema.json`, so every run records a per-category roll-up and per-tool session-share rows as `$scout_structured_output` events — a chartable series for numbers that exist nowhere else (struggle share, problem-tool counts, the regime, and what the scout decided per category), with `references/metrics-dashboard.md` as the consumption recipe.
- `signals-scout-tasks/` — delivery + demand watcher for the Tasks product (the agent work items a project runs), read from the `system.tasks` / `system.task_runs` system tables. Two lenses in one body: **delivery health** every run (runs failing, clustered by repository and by `error_message` prefix) and a ~weekly gated **demand** pass over human-authored tasks (recurring asks that point at a product gap). Its discriminator is failure concentration × spread — a cluster's failure rate over meaningful volume, with runs ÷ distinct tasks separating a _systemic_ defect in a shared path (ratio ≈ 1 across many tasks) from a _retry storm_ (ratio ≫ 1 over one or two stuck tasks), which is the single biggest false positive on this surface. Two exclusions are load-bearing: it always drops `origin_product = 'signals_scout'` (the fleet's own run containers, which the `internal` flag does _not_ filter and which can outnumber every real origin), and the demand lens reads human origins only, so the inbox's own throughput can never register as user demand. On the **report channel** (`emit_report` / `edit_report`): one report per cluster (a repository or a failure class), with the demand lens held to a higher bar — its default outcome is compounding scratchpad memory, and it files only once a theme crosses into something actionable. Bundles `references/queries.md`, a cookbook validated against real task data (it also tells the scout to probe `stage` before resting a lens on it). Task text is treated as untrusted prompt material, never instructions.
- `signals-scout-conversations/` — support-delivery health watcher for the Conversations (support inbox) product. Reads the `$conversation_*` ticket-lifecycle events for operational regressions: SLA breach-rate steps on team replies (`sla_active` / `sla_breached`), first-response latency blowouts (ticket-created → first team reply), backlog inflow-vs-resolution imbalance, and channel / assignment / priority concentration. Its discriminator is a _rate against a volume-stable denominator, per operational dimension, stepping away from its own trailing baseline while ticket volume holds_ — a breach share / response latency / inflow-minus-resolution delta moving on steady volume is signal, a raw count that tracks inbound volume is baseline — always gated by a minimum-volume guard. On the **report channel** (`emit_report` / `edit_report`): files each dated, dimension-named regression as a 1:1 inbox report, editing the live report while the regression persists. It is the operational complement to the per-ticket emission path: the emission pipeline (`source_product="conversations"`) already reads each ticket thread and fires a per-ticket _product-feedback_ signal (bugs / feature requests / usability), so this scout never re-surfaces individual ticket content — it owns the aggregate throughput / SLA / backlog / routing shape the one-ticket-at-a-time emitter can't see, and (reading analytics events) it works whether or not that emission source is enabled.
Expand Down
Loading
Loading