Skip to content

feat(signals): add a scout that watches the organization's own bill - #137999

Merged
trunk-io[bot] merged 2 commits into
masterfrom
posthog/signals-scout-billing-weekly
Nov 18, 2026
Merged

trunk-io[bot] merged 2 commits into
masterfrom
posthog/signals-scout-billing-weekly

Conversation

@posthog

@posthog posthog Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem

  • A customer learns that PostHog usage multiplied only when the invoice arrives, because no scout watches the organization's own bill.

Origin

  • Team member
  • First signal: 2026-11-18
  • Inbox report: open
  • Task started by: a person, from the inbox

Changes

  • New canonical scout, signals-scout-billing. It reports a usage step worth real dollars, or projected spend, a spending limit or a trial end that needs attention.
  • Meters stepping while people hold flat give one instrumentation-change report.
  • A report states one move's dollar delta and names the owning product scout.
  • A weekly pre-check also fires when events double.
  • Without billing access, it files one report naming the fix and scores event volume instead.
  • It supersedes #86516 and fixes its open review findings.

How did you test this code?

Test rationale: a new test fails when a billing usage type lacks a row in the scout's map. The existing quiet-surface test covers the pre-check.

  • Not run: the billing tools live, because both returned a permission error. Not run: a real scout run.

Release status

  • This change is behind a feature flag and is not available to users

Docs update

  • None.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: PostHog Desktop, Claude Opus 5.5 (claude-opus-5-5)

Skills: /writing-tests, /editing-agents-md, /writing-code-comments, /writing-pr-descriptions

Security review: needed

More context for AI agents
  • Session: PostHog Desktop task from an inbox report; PostHog MCP, gh, local datastores for tests.
  • Decisions: stayed on the three billing tools the report names, not the beta billing-forecast-get / billing-limits-get.
  • Decisions: the pre-check mode column (weekly / events_doubled) replaces a boolean, so scout_precheck_evaluated records the branch.
  • Decisions: catch-up cap is 21 days, because a failed run still advances {since} by a week.
  • Follow-ups: run billing-overview-get live and one real scout run before wide release.
  • Follow-ups: gate report, memory and run log visibility by billing access in code, before the scout is enabled by default.
  • Decisions: no call breaks usage down by project, so a run never reads another project's name or usage.
  • Later commit: 72628f541114 added the no-access fallback, written with Claude Code, Claude Fable 5.2 (claude-fable-5-2).
  • Read closely: the pre-check in SKILL.md frontmatter, and the pricing steps in references/usage-types.md.

Created with PostHog Desktop from an inbox report

Adds signals-scout-billing, a canonical scout that files an inbox report when this project's billed usage steps or the organization's projected spend, spending limits or trial end need attention, and hands root cause to the owning product scout.

Ships a weekly pre-check that also fires when events double, a reference mapping every billing usage type to its product key and owning scout, the roster entry, and a test that keeps the usage-type map in step with ee/billing/billing_types.py.

Generated-By: PostHog Desktop
Task-Id: 50726a89-26bf-4766-876a-f5f34af6b133
@posthog posthog Bot added the self-driving label Nov 18, 2026
@posthog
posthog Bot marked this pull request as ready for review November 18, 2026 13:59
@trunk-io

trunk-io Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

😎 Merged successfully - details.

@stamphog
stamphog Bot requested a review from a team November 18, 2026 13:59
@posthog
posthog Bot requested a review from andrewm4894 November 18, 2026 13:59
@parameterai

parameterai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Risk: Medium · 1 medium

This PR ships a new canonical Signals scout, signals-scout-billing, as markdown skill content (SKILL.md, a usage-type reference, AGENTS.md docs) plus a vocabulary test; no production Python changes beyond the test. The pre-check it ships rides the existing harness machinery safely (AST-bound placeholders, read-only, acting-user scoped). The risk that matters: org billing figures flow into team-scoped reports, scratchpad, and Slack with no billing-access gate in code — only skill prose limits what is disclosed.

Sentinel reviewed c89f42b · Review settings

@posthog-security-review-bot posthog-security-review-bot Bot added the security-review Request a security review label Nov 18, 2026
@posthog-security-review-bot

posthog-security-review-bot Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

✅ Security review complete — c89f42bec521

2 findings posted as a review, in 9 min.

Add the security-review label to run again on the latest commit.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

🤖 CI report

⚠️ Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

✅ Duplication (Python) — clean

New Python code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

✅ Duplication (TypeScript) — clean

New TypeScript code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

@posthog-security-review-bot posthog-security-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agent-driven security review - findings inline.

### Lane 2 — the trajectory (every run, one call)

`billing-overview-get` carries the forecast. Read the field mechanics in [`references/usage-types.md`](references/usage-types.md) before quoting a number — the wrong projection field overstates the invoice, and the consequence of crossing a limit is not as certain as it looks.
The response is large (about 70 KB on a modest org: half `available_product_features`, a third `products[].tiers` and `products[].addons`), so make **one** call per run and pull only `billing_period`, `custom_limits_usd`, `discount_percent`, `discount_amount_usd`, the four `projected_total_*` fields, `trial` / `free_trial_until`, and — for each product **and each entry in its `addons[]`**, since add-on meters like identified events, mobile recordings, and 30-day log retention carry their own tiers there — `type`, `usage_key`, `subscribed`, `tiers`, `free_allocation`, `current_usage`, `usage_limit`, `projected_usage`, and `projected_amount_usd`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Billing scout responses are exposed through run logs without billing authorization

medium authz (confidence 4/5)

Details

Latent: enabling this new scout with a billing-authorized acting user makes it retrieve the organization's limits, discounts, tiers and projected totals on every run. billing-overview-get calls /api/billing/ and retains these fields (services/mcp/src/tools/generated/billing.ts:69); that endpoint checks the token's billing scope and acting user's billing access. However, tool responses enter rawOutput (packages/agent/packages/agent/src/adapters/claude/conversion/sdk-to-acp.ts:462) and are persisted by SessionLogWriter (session-log-writer.ts:321,460). Credential redaction does not remove billing figures. The scout profile inherits read_without_channel=Read.TEAM (products/signals/backend/task_profiles.py:58,142), and TaskRunViewSet.logs checks task visibility and task:read, not billing access (products/tasks/backend/presentation/views/api.py:2341,3672). Keeping totals out of report prose and scratchpad memory therefore does not protect the tool results, even on a quiet run that emits no report.

Impact — after an owner runs the scout, an attacker holding a project-scoped API key with only signal_scout:read and task:read can call GET /api/projects/{team}/signals/scout/runs/?skill_name=signals-scout-billing to obtain task_id / task_run_id, then GET /api/projects/{team}/tasks/{task_id}/runs/{task_run_id}/logs/ to read the billing responses. That credential cannot call the billing endpoint directly. The dollar deltas deliberately copied into reports are also served under inbox access rather than billing access.

Fix — add an enforced billing-sensitive audience before enabling this scout: require the appropriate billing scope and user billing permission on its run output and linked task logs/streams/artifacts, and apply equivalent authorization to billing-bearing reports and memory. Prompt-only omissions or hiding the billing page do not enforce this boundary.

Agent-driven security review. The data flow and isolated read-policy probes were verified; a live scout and HTTP reproduction were not run.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid, and not fixed in this PR. Tool responses in run logs sit behind task access, not billing access, and nothing in a skill file can change that. The scout registers disabled on every project, so no run holds billing data until a person with billing access turns it on. Gating run output, reports and memory by billing access in code is the follow-up, and it should land before this scout is enabled by default.

A co-moving group is **one report**: lead with the summed impact, list each meter with its own before and after volume, state that the number of people held, and hand off to the scout that owns the largest meter.
These are decisions about someone's spend, not code fixes: `actionability=requires_human_input`, and leave `priority` / `repository` unset.
**Do not attach `charts`.** Billing series come from a REST tool, not HogQL, so there is no query node that reproduces them — put the numbers in the prose, where a Slack reader gets them too.
**Write for a project-wide audience.** Reports and scratchpad entries are team-scoped, so anyone who can read the inbox reads your billing numbers whether or not the billing pages would let them in — a report is not a place to restate the organization's billing state.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The billing scout reads billing data with an owner's or admin's access and writes it into team inbox reports and scratchpad entries that any project member can read, so it gets around the billing permission check.

medium authz (confidence 4/5)

Details

The new canonical scout signals-scout-billing calls billing-overview-get, billing-spend-get and billing-usage-get with the run's acting user. It then writes what it finds into surfaces that have no billing gate. The only thing that keeps billing figures out of those surfaces is prose in this file: Write for a project-wide audience (this line), Memory holds volumes, dates, and verdicts, never billing detail (line 240), and Do not cache it (line 223). The PR body lists "gate report visibility by billing access in code" as a follow-up, so no control in code covers this yet.

The code path:

  • Who the run acts as. skill_loader.resolve_scout_acting_user_id (skill_loader.py:193-222) resolves a pristine canonical scout to the config's enabled_by/created_by. When neither is set, as with a config that register_missing_configs auto-created, it falls back to resolve_acting_user_id_for_team (temporal/agentic/init.py:105-119). That fallback picks the GitHub integration creator, else the organization's earliest membership, which is usually the owner. The scout roster entry tells admins to "enable it on one project per organization", so the expected setup is a scout running as an owner or admin.
  • What that user can read. BillingViewset.list (ee/api/billing.py:703-711) needs user_has_billing_access: owners always, admins only when owner-only billing is off, members never. Every scout token already carries billing:read, through MCP_READ_SCOPES in posthog/temporal/oauth.py. For a user with full billing access, _scoped_team_ids_for_usage_spend_request (ee/api/billing.py:1424-1428) returns None. A usage or spend call without team_ids therefore spans every project in the organization, including private ones the token's scoped_teams does not cover. The skill makes calls of this kind: the lag probe at line 128, and the per-project attribution of ai_credits_used_in_period at usage-types.md:165.
  • Who reads the output. Inbox reports are readable by anyone with task viewer access on the project (report_access.may_read_reports, report_access.py:6-7; the inbox-scoped report viewset). The scratchpad list needs only signal_scout:read (scout_harness/views.py:2056-2057). Neither checks billing access. The skill also writes for Slack readers on purpose ("put the numbers in the prose, where a Slack reader gets them too").

Even when the model follows every instruction, the output is meant to include data the billing endpoints withhold: each move's discount-adjusted dollar delta (lines 251-262), and from the trajectory lane (lines 204-225), the organization's projected spend against last period, a product nearing a spending limit, and a post-trial charge forecast. The overview behind the trajectory lane is limited to owners, plus admins when owner-only billing is off. It is not part of the members' read-only usage and spend grant. The model also reads text that members can write: project names in label/team_id_options, other reports' titles and summaries via inbox-reports-list, and scratchpad entries. Text there that redirects it can make it write invoice totals, discounts, credit balance and custom limits verbatim, because the only barrier is the instruction not to.

Impact — Organization O has owner Alice and a plain member, Bob. Bob gets 403 BILLING_ACCESS_DENIED_MESSAGE from GET /api/billing/. Alice turns on signals-scout-billing for project P, or the coordinator auto-seeds it and it falls back to her. Its runs act as Alice. Bob opens P's inbox and reads reports such as "Logs ingestion up 9× since Aug 12 — adds ~$180 to this period", plus a trajectory report on the organization's spend trend and spending-limit risk. That is billing state PostHog's own permission check denies him. If the report is routed to Slack, everyone in that channel sees it too. The same reach covers other projects' usage, including private ones, through the org-wide breakdown calls.

Fix — Enforce this in code rather than in the prompt. Options: (1) restrict reports and scratchpad entries authored by this scout, or by any run whose token read a billing endpoint, to users who pass user_has_billing_access, with no Slack routing; (2) keep the scout in withheld_skills until that gate ships; or (3) remove the overview lane and run the step lane with a credential limited to member-level usage on team_ids=[team_id], so the scout never holds more billing reach than its audience. Remove the org-wide ["type","team"] probe and the by-project AI-credit attribution in every case.

Agent-driven security review. The code path is traced end to end. Not visible from the repo: how the production signals-scout flag (allowlist, withheld_skills) seeds this scout, and whether /api/billing/ resolves the scout token's project for a given acting user. view.team is the user's current team, which always matches in a single-project organization.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid. Two parts are fixed in 72628f5: the batch-lag probe no longer breaks usage down by project, and the AI credits row now reads only this project's series, so a run never asks for another project's name or usage. The rest is accepted for now: a report states the dollar delta of one move to inbox readers, with the scout registered disabled on every project. Gating report and memory reads by billing access in code is the follow-up before it is enabled by default.

@posthog-security-review-bot posthog-security-review-bot Bot removed the security-review Request a security review label Nov 18, 2026
_validate_records([StructuredOutputRecord(payload=rejected)], schema)


class TestShippedBillingScoutVocabulary:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Org billing figures reach project members without billing access, enforced only by skill prose

The shipped signals-scout-billing skill (in products/signals/skills/signals-scout-billing/) reads the organization's bill under an acting user with billing:read, then publishes dollar deltas, spending-limit proximity, and trial-end warnings into inbox reports, the scratchpad, and Slack — surfaces gated only by team membership (ScoutCanonicalTeamAccessPermission, products/signals/backend/presentation/scout_permissions.py:13), never by billing access. The skill's own text admits it: "anyone who can read the inbox reads your billing numbers whether or not the billing pages would let them in" (SKILL.md:257). The sole control is LLM instruction-following ("carry only the dollar delta", SKILL.md:258; "memory holds volumes, dates, and verdicts, never billing detail", SKILL.md:240), and the PR lists "gate report visibility by billing access in code" as an undone follow-up — so an org admin's billing read is summarized into team- and Slack-visible facts for members the billing pages deny.

How:

  1. An org admin enables the scout (or is its acting user), so the billing tools resolve.
  2. Any run files a report such as "adds ~$180 to this period" plus limit/trial warnings.
  3. A project member with no billing access reads the inbox report or the Slack delivery — no code check exists.
  4. Prose is the only filter, so a drifting or injected run (e.g. a customer-controlled project name pushing it to restate totals) has nothing in code to stop it.

Fix: gate this scout's report/scratchpad reads (or its emission content) on billing access in code, as the PR's own follow-up already plans.


React with 👍 if useful or 👎 if not

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid, and accepted for now rather than fixed here. The scout registers disabled on every project, so a person with billing access has to turn it on before any report exists. The code gate on report and memory reads is the follow-up before it is enabled by default. 72628f5 also removes the two calls that could read other projects' usage.

…olume

A billing scout whose acting user holds no billing access closed out
empty on every run, so it looked healthy while watching nothing. It now
files one report telling the owner what is missing and how to fix it,
and scores the project's own event volume as a stand-in.

The batch-lag probe no longer breaks usage down by project, so a run
never reads another project's name or usage.

Unverified: docs
@andrewm4894

Copy link
Copy Markdown
Member

/trunk merge

@trunk-io
trunk-io Bot merged commit 0bc6e14 into master Nov 18, 2026
295 checks passed
@trunk-io
trunk-io Bot deleted the posthog/signals-scout-billing-weekly branch November 18, 2026 17:20
@deployment-status-posthog

deployment-status-posthog Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ❔ Unknown
prod-us ✅ Deployed 2026-11-18 18:30 UTC Run
prod-eu ✅ Deployed 2026-11-18 18:27 UTC Run

This table shows the posthog release only. Hogstation shows every service that this PR reaches, Kargo services included.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant