Repository navigation
feat(signals): add a scout that watches the organization's own bill - #137999
Conversation
Adds signals-scout-billing, a canonical scout that files an inbox report when this project's billed usage steps or the organization's projected spend, spending limits or trial end need attention, and hands root cause to the owning product scout. Ships a weekly pre-check that also fires when events double, a reference mapping every billing usage type to its product key and owning scout, the roster entry, and a test that keeps the usage-type map in step with ee/billing/billing_types.py. Generated-By: PostHog Desktop Task-Id: 50726a89-26bf-4766-876a-f5f34af6b133
|
😎 Merged successfully - details. |
|
Risk: Medium · 1 medium This PR ships a new canonical Signals scout, Sentinel reviewed |
|
✅ Security review complete — 2 findings posted as a review, in 9 min. Add the |
🤖 CI report
|
| ### Lane 2 — the trajectory (every run, one call) | ||
|
|
||
| `billing-overview-get` carries the forecast. Read the field mechanics in [`references/usage-types.md`](references/usage-types.md) before quoting a number — the wrong projection field overstates the invoice, and the consequence of crossing a limit is not as certain as it looks. | ||
| The response is large (about 70 KB on a modest org: half `available_product_features`, a third `products[].tiers` and `products[].addons`), so make **one** call per run and pull only `billing_period`, `custom_limits_usd`, `discount_percent`, `discount_amount_usd`, the four `projected_total_*` fields, `trial` / `free_trial_until`, and — for each product **and each entry in its `addons[]`**, since add-on meters like identified events, mobile recordings, and 30-day log retention carry their own tiers there — `type`, `usage_key`, `subscribed`, `tiers`, `free_allocation`, `current_usage`, `usage_limit`, `projected_usage`, and `projected_amount_usd`. |
There was a problem hiding this comment.
Billing scout responses are exposed through run logs without billing authorization
Details
Latent: enabling this new scout with a billing-authorized acting user makes it retrieve the organization's limits, discounts, tiers and projected totals on every run. billing-overview-get calls /api/billing/ and retains these fields (services/mcp/src/tools/generated/billing.ts:69); that endpoint checks the token's billing scope and acting user's billing access. However, tool responses enter rawOutput (packages/agent/packages/agent/src/adapters/claude/conversion/sdk-to-acp.ts:462) and are persisted by SessionLogWriter (session-log-writer.ts:321,460). Credential redaction does not remove billing figures. The scout profile inherits read_without_channel=Read.TEAM (products/signals/backend/task_profiles.py:58,142), and TaskRunViewSet.logs checks task visibility and task:read, not billing access (products/tasks/backend/presentation/views/api.py:2341,3672). Keeping totals out of report prose and scratchpad memory therefore does not protect the tool results, even on a quiet run that emits no report.
Impact — after an owner runs the scout, an attacker holding a project-scoped API key with only signal_scout:read and task:read can call GET /api/projects/{team}/signals/scout/runs/?skill_name=signals-scout-billing to obtain task_id / task_run_id, then GET /api/projects/{team}/tasks/{task_id}/runs/{task_run_id}/logs/ to read the billing responses. That credential cannot call the billing endpoint directly. The dollar deltas deliberately copied into reports are also served under inbox access rather than billing access.
Fix — add an enforced billing-sensitive audience before enabling this scout: require the appropriate billing scope and user billing permission on its run output and linked task logs/streams/artifacts, and apply equivalent authorization to billing-bearing reports and memory. Prompt-only omissions or hiding the billing page do not enforce this boundary.
Agent-driven security review. The data flow and isolated read-policy probes were verified; a live scout and HTTP reproduction were not run.
There was a problem hiding this comment.
Valid, and not fixed in this PR. Tool responses in run logs sit behind task access, not billing access, and nothing in a skill file can change that. The scout registers disabled on every project, so no run holds billing data until a person with billing access turns it on. Gating run output, reports and memory by billing access in code is the follow-up, and it should land before this scout is enabled by default.
| A co-moving group is **one report**: lead with the summed impact, list each meter with its own before and after volume, state that the number of people held, and hand off to the scout that owns the largest meter. | ||
| These are decisions about someone's spend, not code fixes: `actionability=requires_human_input`, and leave `priority` / `repository` unset. | ||
| **Do not attach `charts`.** Billing series come from a REST tool, not HogQL, so there is no query node that reproduces them — put the numbers in the prose, where a Slack reader gets them too. | ||
| **Write for a project-wide audience.** Reports and scratchpad entries are team-scoped, so anyone who can read the inbox reads your billing numbers whether or not the billing pages would let them in — a report is not a place to restate the organization's billing state. |
There was a problem hiding this comment.
The billing scout reads billing data with an owner's or admin's access and writes it into team inbox reports and scratchpad entries that any project member can read, so it gets around the billing permission check.
Details
The new canonical scout signals-scout-billing calls billing-overview-get, billing-spend-get and billing-usage-get with the run's acting user. It then writes what it finds into surfaces that have no billing gate. The only thing that keeps billing figures out of those surfaces is prose in this file: Write for a project-wide audience (this line), Memory holds volumes, dates, and verdicts, never billing detail (line 240), and Do not cache it (line 223). The PR body lists "gate report visibility by billing access in code" as a follow-up, so no control in code covers this yet.
The code path:
- Who the run acts as.
skill_loader.resolve_scout_acting_user_id(skill_loader.py:193-222) resolves a pristine canonical scout to the config'senabled_by/created_by. When neither is set, as with a config thatregister_missing_configsauto-created, it falls back toresolve_acting_user_id_for_team(temporal/agentic/init.py:105-119). That fallback picks the GitHub integration creator, else the organization's earliest membership, which is usually the owner. The scout roster entry tells admins to "enable it on one project per organization", so the expected setup is a scout running as an owner or admin. - What that user can read.
BillingViewset.list(ee/api/billing.py:703-711) needsuser_has_billing_access: owners always, admins only when owner-only billing is off, members never. Every scout token already carriesbilling:read, throughMCP_READ_SCOPESin posthog/temporal/oauth.py. For a user with full billing access,_scoped_team_ids_for_usage_spend_request(ee/api/billing.py:1424-1428) returnsNone. A usage or spend call withoutteam_idstherefore spans every project in the organization, including private ones the token'sscoped_teamsdoes not cover. The skill makes calls of this kind: the lag probe at line 128, and the per-project attribution ofai_credits_used_in_periodat usage-types.md:165. - Who reads the output. Inbox reports are readable by anyone with
taskviewer access on the project (report_access.may_read_reports, report_access.py:6-7; theinbox-scoped report viewset). The scratchpad list needs onlysignal_scout:read(scout_harness/views.py:2056-2057). Neither checks billing access. The skill also writes for Slack readers on purpose ("put the numbers in the prose, where a Slack reader gets them too").
Even when the model follows every instruction, the output is meant to include data the billing endpoints withhold: each move's discount-adjusted dollar delta (lines 251-262), and from the trajectory lane (lines 204-225), the organization's projected spend against last period, a product nearing a spending limit, and a post-trial charge forecast. The overview behind the trajectory lane is limited to owners, plus admins when owner-only billing is off. It is not part of the members' read-only usage and spend grant. The model also reads text that members can write: project names in label/team_id_options, other reports' titles and summaries via inbox-reports-list, and scratchpad entries. Text there that redirects it can make it write invoice totals, discounts, credit balance and custom limits verbatim, because the only barrier is the instruction not to.
Impact — Organization O has owner Alice and a plain member, Bob. Bob gets 403 BILLING_ACCESS_DENIED_MESSAGE from GET /api/billing/. Alice turns on signals-scout-billing for project P, or the coordinator auto-seeds it and it falls back to her. Its runs act as Alice. Bob opens P's inbox and reads reports such as "Logs ingestion up 9× since Aug 12 — adds ~$180 to this period", plus a trajectory report on the organization's spend trend and spending-limit risk. That is billing state PostHog's own permission check denies him. If the report is routed to Slack, everyone in that channel sees it too. The same reach covers other projects' usage, including private ones, through the org-wide breakdown calls.
Fix — Enforce this in code rather than in the prompt. Options: (1) restrict reports and scratchpad entries authored by this scout, or by any run whose token read a billing endpoint, to users who pass user_has_billing_access, with no Slack routing; (2) keep the scout in withheld_skills until that gate ships; or (3) remove the overview lane and run the step lane with a credential limited to member-level usage on team_ids=[team_id], so the scout never holds more billing reach than its audience. Remove the org-wide ["type","team"] probe and the by-project AI-credit attribution in every case.
Agent-driven security review. The code path is traced end to end. Not visible from the repo: how the production signals-scout flag (allowlist, withheld_skills) seeds this scout, and whether /api/billing/ resolves the scout token's project for a given acting user. view.team is the user's current team, which always matches in a single-project organization.
There was a problem hiding this comment.
Valid. Two parts are fixed in 72628f5: the batch-lag probe no longer breaks usage down by project, and the AI credits row now reads only this project's series, so a run never asks for another project's name or usage. The rest is accepted for now: a report states the dollar delta of one move to inbox readers, with the scout registered disabled on every project. Gating report and memory reads by billing access in code is the follow-up before it is enabled by default.
| _validate_records([StructuredOutputRecord(payload=rejected)], schema) | ||
|
|
||
|
|
||
| class TestShippedBillingScoutVocabulary: |
There was a problem hiding this comment.
Org billing figures reach project members without billing access, enforced only by skill prose
The shipped signals-scout-billing skill (in products/signals/skills/signals-scout-billing/) reads the organization's bill under an acting user with billing:read, then publishes dollar deltas, spending-limit proximity, and trial-end warnings into inbox reports, the scratchpad, and Slack — surfaces gated only by team membership (ScoutCanonicalTeamAccessPermission, products/signals/backend/presentation/scout_permissions.py:13), never by billing access. The skill's own text admits it: "anyone who can read the inbox reads your billing numbers whether or not the billing pages would let them in" (SKILL.md:257). The sole control is LLM instruction-following ("carry only the dollar delta", SKILL.md:258; "memory holds volumes, dates, and verdicts, never billing detail", SKILL.md:240), and the PR lists "gate report visibility by billing access in code" as an undone follow-up — so an org admin's billing read is summarized into team- and Slack-visible facts for members the billing pages deny.
How:
- An org admin enables the scout (or is its acting user), so the billing tools resolve.
- Any run files a report such as "adds ~$180 to this period" plus limit/trial warnings.
- A project member with no billing access reads the inbox report or the Slack delivery — no code check exists.
- Prose is the only filter, so a drifting or injected run (e.g. a customer-controlled project name pushing it to restate totals) has nothing in code to stop it.
Fix: gate this scout's report/scratchpad reads (or its emission content) on billing access in code, as the PR's own follow-up already plans.
React with 👍 if useful or 👎 if not
There was a problem hiding this comment.
Valid, and accepted for now rather than fixed here. The scout registers disabled on every project, so a person with billing access has to turn it on before any report exists. The code gate on report and memory reads is the follow-up before it is enabled by default. 72628f5 also removes the two calls that could read other projects' usage.
…olume A billing scout whose acting user holds no billing access closed out empty on every run, so it looked healthy while watching nothing. It now files one report telling the owner what is missing and how to fix it, and scores the project's own event volume as a stand-in. The batch-lag probe no longer breaks usage down by project, so a run never reads another project's name or usage. Unverified: docs
|
/trunk merge |
Deploy status
This table shows the |
Problem
Origin
Changes
signals-scout-billing. It reports a usage step worth real dollars, or projected spend, a spending limit or a trial end that needs attention.How did you test this code?
Test rationale: a new test fails when a billing usage type lacks a row in the scout's map. The existing quiet-surface test covers the pre-check.
Release status
Docs update
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Agent: PostHog Desktop, Claude Opus 5.5 (
claude-opus-5-5)Skills:
/writing-tests,/editing-agents-md,/writing-code-comments,/writing-pr-descriptionsSecurity review: needed
More context for AI agents
gh, local datastores for tests.billing-forecast-get/billing-limits-get.modecolumn (weekly/events_doubled) replaces a boolean, soscout_precheck_evaluatedrecords the branch.{since}by a week.billing-overview-getlive and one real scout run before wide release.72628f541114added the no-access fallback, written with Claude Code, Claude Fable 5.2 (claude-fable-5-2).SKILL.mdfrontmatter, and the pricing steps inreferences/usage-types.md.Created with PostHog Desktop from an inbox report