Skip to content

feat(signals): add canonical billing scout - #86516

Closed
andrewm4894 wants to merge 16 commits into
masterfrom
posthog/signals-scout-billing
Closed

andrewm4894 wants to merge 16 commits into
masterfrom
posthog/signals-scout-billing

Conversation

@andrewm4894

@andrewm4894 andrewm4894 commented Aug 20, 2026 •

Copy link
Copy Markdown
Member

Problem

An organization learns that its PostHog bill jumped when the invoice arrives. No scout watches the organization's own bill (origin thread, recent prompt).

Changes

  • A new canonical scout, signals-scout-billing, files an inbox report when the bill moves and names the product scout that owns the cause.
  • It scores daily usage per meter against a same-weekday baseline, and reports only steps worth real dollars.
  • Meters stepping together with flat distinct people are one report, not traffic growth.
  • A default pre-check starts a scheduled run about weekly, or sooner when event volume doubles.
  • Mechanical: one roster entry.

Note

The scout seeds disabled on new projects under the fleet's current seed allowlist. A person turns it on.
A report shows the dollar delta of one move to everyone who can read the project inbox, including members without billing access. Accepted for now.

How did you test this code?

  • The usage, spend, people-check and pre-check queries ran by hand against a development project.
  • Not run: the overview call, and a real scout run.

Test rationale: None added. The diff is a skill body with no code path.

Release status

  • No feature flag controls this change

Docs update

  • None.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: PostHog Slack app, Claude Opus 5; Claude Code, Fable 5.2

Skills: /authoring-scouts, /writing-pr-descriptions

Security review: needed

More context for AI agents
  • Session: Claude Code with the PostHog MCP, reopened from a stale close.
  • Decisions: Array parameters replace the JSON-string form, because the billing tools now take real arrays.
  • Decisions: The access gate no longer names a feature flag. The three tools the scout uses are ungated.
  • Decisions: Five usage types added since August are mapped, including the billed logs and traces meter.
  • Decisions: Spend comparisons anchor on pricing_reset_dates, which can differ from the renewal date.
  • Decisions: The concentration gate no longer drops co-moving meters on sight. One events-per-person query separates growth from instrumentation.
  • Decisions: Scout memory holds volumes and verdicts only. Tier prices and invoice totals are re-read each run, after a review finding.
  • Follow-ups: Gate billing report content by billing access in code.
  • Follow-ups: The billing service's spike detection could start a run directly. Today that message only reaches account managers.
  • Follow-ups: The newer per-period billing tools return smaller payloads than the overview. Move the trajectory lane to them once they leave beta.
  • Read closely: SKILL.md, the "Machine-shaped?" gate and the memory rule. Security: reports put billing deltas in a team-scoped inbox.

Created with PostHog from a Slack thread

Adds `signals-scout-billing`, a canonical Signals scout that watches an
organization's own PostHog bill through the customer-facing billing read
tools (`billing-overview-get` / `billing-usage-get` / `billing-spend-get`).

Two lanes per run: a step lane scoring each (usage type x project) pair's
daily volume against its own same-weekday trailing baseline, and a
trajectory lane reading projected period spend, spending limits, and tier
position. Materiality is measured in dollars at the org's current tier
position rather than in percent, and the body treats tier resets at the
period boundary as the surface's defining false positive.

Bundles `references/usage-types.md` for the usage-type to owning-scout
routing table and the tier/period mechanics.

Generated-By: PostHog Desktop
Task-Id: e9d7c6c7-1798-49be-80e4-8ddcf8a7734c
@andrewm4894 andrewm4894 self-assigned this Aug 20, 2026
@trunk-io

trunk-io Bot commented Aug 20, 2026 •

Copy link
Copy Markdown

🚫 This pull request was removed from the merge queue because it was pushed to by @andrewm4894. Please re-submit it in order to merge. See more details here.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

@github-actions

Copy link
Copy Markdown
Contributor

Hey @andrewm4894! 👋

It looks like your git author email on this PR isn't your @posthog.com address (andrewm4894@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

  • Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
  • Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

@andrewm4894
andrewm4894 marked this pull request as ready for review August 20, 2026 14:15
@pr-assigner-resolver-posthog
pr-assigner-resolver-posthog Bot requested a review from a team August 20, 2026 14:16
@github-actions

Copy link
Copy Markdown
Contributor

🤖 CI report

⚠️ Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

@greptile-apps

greptile-apps Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Reviews (1): Last reviewed commit: "feat(signals): add canonical billing sco..." | Re-trigger Greptile

chatgpt-codex-connector[bot]

This comment was marked as outdated.

Addresses review feedback, each item verified against the implementation.

Field semantics:
- Derive the free threshold from the tier schedule, not `free_allocation`.
  A subscribed product can ship `free_allocation: 0` with its first tier
  free, so the old instruction classified wholly free usage as billable.
- Quote a `*_with_limit*` projection when forecasting the invoice. The
  uncapped field is a hypothetical the customer is never billed.
- Describe crossing a spending limit as a possibility. `never_drop_data`
  changes the outcome and the MCP tool strips it from the response.
- Check the legacy `free_trial_until` alongside the `trial` object.
- Branch on `breakdown_type` rather than assuming a two-element
  `breakdown_value`; type-only series carry one element and team ids
  come back as strings.
- Price materiality at the current marginal tier and keep
  `billing-spend-get` inside one period, so a tier reset cannot clear
  the gate on arithmetic alone.
- Store billing period boundaries with the cached period actual and
  re-derive once the period advances.

Scoring and scope:
- State the band numerically: robust z >= 3.5, a 50% fallback when MAD
  is zero, a 100-units/day volume floor, +/-20% concentration tolerance,
  and a materiality floor of the greater of $20 and 2% of last period's
  invoice.
- Pin the step lane to this project's `team_ids` so multiple enabled
  projects in one org cannot file duplicate reports; note that the
  trajectory lane stays org-wide and release belongs on one project per
  organization.
- Page the scratchpad with a `keys_only` scan and exact key re-reads
  rather than relying on the 20-row default.
- Define an evidence-backed reviewer ladder and file unrouted rather
  than guess, since no runtime source names the billing owner.

Also reformats the markdown with oxfmt to fix the failing frontend
formatting check.

Generated-By: PostHog Desktop
Task-Id: e9d7c6c7-1798-49be-80e4-8ddcf8a7734c

Copy link
Copy Markdown
Member Author

Worked through all 11 Codex findings. Verified each against the implementation first — every one held up, so all are addressed except the P1 rollout gate, which cannot be a code change (replied on that thread: withheld_skills lives only in the signals-scout flag payload).

Field semantics that were wrong and would have produced false or overstated reports:

  • Free tier. Was reading free_allocation. calculateFreeTier in billing-utils.ts uses the first zero-priced tier's up_to for subscribed products, and a subscribed product can ship free_allocation: 0 with its first million units free — so the old rule would have filed a billing alert on entirely free usage. Now derived from the tier schedule, with free_allocation as the unsubscribed fallback.
  • Projection field. Was quoting the uncapped projected_total_amount_usd_after_discount. BillingSummary.tsx shows *_with_limit_after_discount and tells customers it is the maximum charge, so the old field overstates the invoice wherever a limit applies. Now the capped field forecasts, and the uncapped one is reserved for limit-risk — the gap between them is exactly the usage a limit is about to cut off.
  • Limit consequences. Was asserting data gets dropped. never_drop_data makes quota_limiting.py ignore exceeded limits for non-exempt resources, and tools.yaml strips that field from the response — so the scout genuinely cannot know. Now written as a possibility.
  • Trials. Was checking only trial. free_trial_until is still supported and is checked first in billing-utils.ts, so orgs on the legacy field would never have gotten the warning. Both checked now.
  • breakdown_value. Was assumed to be [usage_type, team_id]. The fixtures in read_billing_tool/test/test_nodes.py show one-element values for both TYPE and TEAM breakdowns, with team ids as strings. Now branches on breakdown_type.
  • Materiality across periods. The 35-day spend window normally crosses a period boundary, and tier resets mean identical usage bills higher early in a period — so it could clear the gate with no usage change at all. Materiality is now priced at the current marginal tier, and billing-spend-get is confined to one period.

Scoring and scope:

  • The band is now numeric — robust z ≥ 3.5 over four same-weekday values, a ≥50% fallback when MAD is zero, a 100-units/day volume floor, ±20% concentration tolerance, and a materiality floor of the greater of $20 and 2% of last period's invoice. That last one is deliberately relative so a small org still hears about small dollars without paging a large one over rounding. Previously "band" was undefined and 1→3 units could read as a step.
  • Duplicate org-wide reports. The step lane now pins team_ids to its own project, so it dedupes by construction and its findings are true as written whatever access the caller has. The trajectory lane is irreducibly org-wide, so that constraint moved to the release gate: one project per organization.
  • Scratchpad paging. keys_only wide scan then exact key= re-reads, with date_to cursor paging — the 20-row default would have dropped dedupe pointers once baselines accumulated.
  • Reviewer routing. Nothing at runtime names the billing owner: scout-members-list has no org role or billing permission, and the overview strips account_owner. Rather than invent a source, the body now defines a ladder (cached → inbox precedent → recent_reviewer_corrections from the project profile, which the harness prompt already treats as the strongest ownership evidence) and files unrouted when none resolves. An unrouted report someone picks up beats a confidently mis-assigned one.

Period rollover also fixed: the cached period actual now stores its billing_period boundaries and re-derives when the period advances, so a rollover cannot silently compare against two-period-old history.

The failing check was markdown formatting — both files reformatted with the catalog-pinned oxfmt 0.57.0; oxfmt --check and markdownlint-cli2 both pass locally.

Still unverified and unchanged from the original description: none of this has run against live billing data. billing-mcp-read-tools is opt-in and not granted in the authoring session, so the scout still needs one scout-run-now and a run inspection before release.

chatgpt-codex-connector[bot]

This comment was marked as outdated.

Dogfooded against the billing read tools on a real organization:
type-only breakdowns return a string breakdown_value, today's bucket
reads zero rather than partial, spend keys series by product rather than
usage type, and usage_limit is the spending limit in units.
chatgpt-codex-connector[bot]

This comment was marked as outdated.

andrewm4894 and others added 2 commits August 20, 2026 16:13
Second review pass, layered on top of the dogfooding commit. Most of
these are consequences of the previous round's fixes rather than of the
original body.

Would have broken every run:
- Pass `breakdowns` / `team_ids` as JSON-encoded strings. The MCP
  handler types all three array params as Zod strings, so an actual
  array is rejected before the request is made.

Self-contradictions introduced by the last review round:
- The usage-only grant was told to run the step lane, which then
  required overview tier data for its mandatory dollar gate. It now
  runs in volume-only mode, reporting just the two shapes decisive
  without a dollar figure - a meter going dark, and a >=5x step holding
  three or more days - and carrying the rest as memory.
- The usage cursor close-out stopped the whole run, so a sparse
  designated project could suppress an org-wide spending-limit or
  trial-expiry warning. It now gates lane 1 only.
- The volume floor tested the baseline median alone, discarding a meter
  rising from near zero to a large paid spike - the cold-start
  instrumentation loop the scout most wants to catch. It now skips only
  when both the baseline and the latest value are under the floor, and
  a zero baseline is handled as a new-meter onset.

Correctness:
- Price a delta as `price(with) - price(without)` across the tier
  schedule, then discount and cap. A flat rate times the delta
  overstates any move that crosses a tier, and a fully discounted or
  already-capped meter can add $0 while clearing the gate. Keeps the
  observed per-tier `current_usage` / `projected_usage` detail for
  finding the marginal tier, and caps against the observed
  units-denominated `usage_limit`.
- Group alias and subset meters into families before the concentration
  test. `group_analytics` is a view of identified-event usage and the
  logs retention tier is the same content, so counting them separately
  let one real move read as a broad traffic story.
- Route `ai_credits_used_in_period` away from the AI-observability
  scout. Those credits are metered from PostHog's own regional
  projects, not the customer's `$ai_*` events, so that scout would
  close out finding nothing. The dogfooded vocabulary table agrees:
  they bill under the `posthog_ai` product, not `llm_analytics`.
- Point the materiality gate at that vocabulary table, since tiers and
  limits are keyed by product rather than by usage type.
- Pass `skill_name` to `scout-runs-list`; the fleet's daily schedules
  push this scout's own last run outside the 20-row default.

Generated-By: PostHog Desktop
Task-Id: e9d7c6c7-1798-49be-80e4-8ddcf8a7734c
…ling data

Fallback order for pricing a step when the overview is denied, truncated,
or lacks tiers; treat every trajectory field as optional; use the flat
materiality floor when no prior-period invoice exists; calibrate on the
first runs and report unknown response shapes through agent-feedback.

Copy link
Copy Markdown
Member Author

Round two addressed, and rebuilt on top of the dogfooding commit rather than over it — 50a1cbef had landed while this was in flight, so all eight fixes are layered on the observed responses. The live findings are intact: the plain-string breakdown_value for ["type"], today's bucket reading zero rather than partial, spend keyed by product, the units-denominated usage_limit, and the three-vocabulary table.

One would have broken every run. breakdowns, team_ids, and usage_types are zod.string() in services/mcp/src/generated/billing/api.ts, so the array literal in the call example is rejected before the request is made. Now spelled out as JSON strings.

Three were self-contradictions the previous round introduced, which is the useful part of a second pass:

  • The usage-only grant was told to run the step lane, which then required overview tier data for its own mandatory dollar gate — so it could only guess (forbidden) or suppress everything. It now runs in volume-only mode: reports the two shapes decisive without a dollar figure (a meter going dark; a ≥5× step holding 3+ days), says plainly that sizing needs billing access, carries the rest as memory.
  • The cursor close-out stopped the whole run, so a sparse designated project could mute an org-wide spending-limit or trial-expiry warning driven by a sibling project. Now gates lane 1 only.
  • The volume floor tested the baseline median alone — which discards a meter going from near-zero to a large paid spike, i.e. exactly the cold-start instrumentation loop this scout exists for. Now skips only when both sides are under the floor, with a zero baseline handled as a new-meter onset.

Pricing was wrong in a way that could clear the gate at $0. delta × marginal rate overstates any move crossing a tier, and says nothing about a full discount or an already-binding cap. Now price(with) − price(without) across the schedule, then discount, then cap — keeping the observed per-tier current_usage for locating the marginal tier and the units-denominated usage_limit for the cap.

Two routing corrections. Alias meters (group_analytics as a view of identified events, the logs retention tier as the same content) are grouped into families before the concentration test, since counting them separately let one real move read as a broad traffic story. And ai_credits_used_in_period no longer routes to the AI-observability scout: _get_teams_with_ai_credits_for_products meters it from PostHog's own regional projects rather than the customer's $ai_* events, so that scout would close out finding nothing — the dogfooded table agrees, billing it under posthog_ai rather than llm_analytics.

scout-runs-list now passes skill_name too; with daily fleet schedules this scout's own last run falls outside the 20-row default.

CI green. Rollout gates unchanged and still outside this diff: add to default_team_config.withheld_skills, and release to one project per organization.

chatgpt-codex-connector[bot]

This comment was marked as outdated.

… persistence pricing

Score a complete all-zero day instead of skipping to the last non-zero
one; keep addons, discount_percent, and free_allocation from the
overview; price a step as it projects to period end; count simultaneous
meter onsets in the concentration gate; forecast post-trial charges only
for active autosubscribe trials.
@veria-ai

veria-ai Bot commented Aug 20, 2026 •

Copy link
Copy Markdown
Contributor

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

Reviews

chatgpt-codex-connector[bot]

This comment was marked as outdated.

…od math

Keep `projected_usage` in the overview pull list so the materiality
counterfactual has its starting unit count. Distinguish batch lag from a real
project outage with project-scoped evidence rather than the org-wide usage
summary. Group the PostHog Desktop aggregate meter with its token and compute
components before the concentration test. Normalize period-over-period spend to
a daily run rate so a longer calendar month is not read as a trajectory
finding. Tell reports to carry only the billing figures the finding needs,
since the inbox is team-scoped.

Generated-By: PostHog Desktop
Task-Id: 4a683539-86ee-427e-ac90-b9178a7fcfa6
@andrewm4894 andrewm4894 added the stamphog Request AI approval (no full review) label Aug 20, 2026 — with PostHog
@andrewm4894

Copy link
Copy Markdown
Member Author

/trunk merge

chatgpt-codex-connector[bot]

This comment was marked as outdated.

@stamphog

stamphog Bot commented Aug 20, 2026 •

Copy link
Copy Markdown

Gates denied: this PR touches billing logic (deny-listed) and is sized above the auto-approvable tier; it also has multiple unresolved reviewer findings (a billing-data-exposure concern and several P2 correctness issues in the pricing/materiality math) that haven't all been confirmed fixed on the current head.

  • chatgpt-codex-connector[bot] reviewed the current head.
  • Deny-list match: billing-related content, and tier classified T2-never — outside auto-approval scope
  • Unresolved security finding from @veria-ai about billing data potentially visible to team members without billing access, with only a documented-but-not-code-level mitigation
  • Several Codex P2 findings on pricing/materiality math posted after the latest fix commit, not yet confirmed addressed on current head
Gate mechanics and policy version
Gate Result
prerequisites ✓ all clear
deny-list ✗ matches: billing
size ✓ 0L, 0F substantive, 444L/3F incl. docs/generated/snapshots — within ceiling
tier ✗ classified as T2-never: T2-never (444L, 3F, single-area, feat)
stamphog 2.0.0b4 .stamphog/policy.yml @ b4b3252 · reviewed head b4b3252

@stamphog stamphog Bot added reviewhog and removed stamphog Request AI approval (no full review) labels Aug 20, 2026
@posthog

posthog Bot commented Aug 20, 2026 •

Copy link
Copy Markdown
Contributor

🦔 ReviewHog reviewed this pull request

Nothing worth raising this time, so here's a calming picture instead:

A happy dog on a sunny path

Resolved comments: 3 declined, 8 already settled, 2 left for you

@stamphog stamphog Bot removed the stamphog Request AI approval (no full review) label Aug 20, 2026
@scheduled-actions-posthog

Copy link
Copy Markdown
Contributor

This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, please remove the stale label – otherwise this will be closed in another week. If you want to permanently keep it open, use the waiting label.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Aug 29, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-18T12:52:03.756876Z 6008c80 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector[bot]

This comment was marked as outdated.

Four fixes from the latest review round on the billing scout:

- Lane 1 now scores every complete day after the cursor, oldest first and capped at 7, so a spike inside a run gap is not retired unscored. Consecutive days of one step stay one report.
- The concentration test only compares against families that clear the volume floor, so a 1 -> 0 wobble no longer rejects a material candidate.
- Delta pricing anchors both scenarios to the service's own projected_usage instead of adding the step on top of a forecast that already contains it, which slid both endpoints up the tier curve and invented impact across a free threshold.
- A step's impact is described as what it adds to this period, not as being above last period; period-over-period stays with the trajectory lane.

Generated-By: PostHog Desktop
Task-Id: 96734564-6ffa-4a67-aaa2-5dcfd10b2e1e

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fcc51f06f7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".


- Projected period spend materially above the last complete period's actual.
**Attribute it, do not require it to be unattributable.** A gradual single-product ramp is the shape lane 1 structurally cannot see — a trailing same-weekday baseline climbs with the ramp, so no day is ever an outlier — and it is exactly what this lane exists for. When one product explains the higher forecast, name that product in the report; only skip when lane 1 already filed the same meter this period.
**Compare daily run rates, not raw totals.** Billing periods are calendar months of different lengths, so a March projection sits ~11% above an unchanged February at an identical daily rate — enough to clear the materiality floor on the calendar alone. Divide each side by its own period's day count, or scale the prior total to the current period's length, before calling the difference surprising.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Exclude flat charges from period-length normalization

When an invoice includes a monthly flat charge, dividing the whole total by each period's day count creates a trajectory change even when the charge is unchanged; for example, a fixed $1,000 invoice appears about 10.7% more expensive per day in February than January. Fresh evidence beyond the earlier period-length fix is the supported nonzero flat_amount_usd in frontend/src/mocks/fixtures/_billing_with_flat_fee.json:464, which the pricing reference explicitly includes, so normalize only usage-proportional charges and compare fixed charges directly.

Useful? React with 👍 / 👎.

The response is `results`: a list of series, each carrying `label`, `dates`, `data`, `breakdown_type`, and `breakdown_value`, plus a top-level `team_id_options` listing every project the caller can see.

**`breakdown_value`'s shape follows `breakdown_type` — branch on it, never index blindly.**
With `["type"]`, `breakdown_type` is `"type"` and `breakdown_value` is a **plain string** (`"recording_count_in_period"`).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Accept singleton arrays for type breakdowns

When billing-usage-get returns the supported singleton-array shape for a type breakdown, this branch expects a string and cannot match the usage type to the vocabulary or tier schedule. The Billing UI explicitly normalizes both string and array values in frontend/src/scenes/billing/billingUsageLogic.ts:42-53,74, and the production-shaped billing test uses ["billable_feature_flag_requests_count_in_period"] with breakdown_type="type"; unwrap a singleton array as well or valid meters fall into the unknown, volume-only path.

AGENTS.md reference: products/signals/skills/AGENTS.md:L57-L57

Useful? React with 👍 / 👎.

- A caller who cannot see a sibling project has no way to tell the two apart. Score the zero day and say in the report that a delayed batch has not been ruled out, rather than dismissing it.

Use the org `usage_summary` only to corroborate, never to decide.
When the day really is lagging, score the last non-zero day but **leave the cursor at the last day you actually scored**, and record the lagging dates separately as `pattern:billing:lag`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid rescoring the cursor date during lag

Fresh evidence after the cursor fix is that, during an ordinary multi-day batch delay, the last nonzero day is usually the date already stored in pattern:billing:cursor, so this fallback scores the same bucket on every scheduled run while leaving the cursor unchanged. If that bucket belongs to a live step episode, line 229 can append the same evidence and running total repeatedly even though edit_report(append_note=...) is explicitly non-idempotent in products/signals/backend/scout_harness/prompt.py:404-407; score the fallback only when it is strictly newer than the cursor, otherwise skip lane 1 and continue the trajectory lane.

AGENTS.md reference: products/signals/skills/AGENTS.md:L57-L57

Useful? React with 👍 / 👎.

### Lane 2 — the trajectory (every run, one call)

`billing-overview-get` carries the forecast. Read the field mechanics in [`references/usage-types.md`](references/usage-types.md) before quoting a number — the wrong projection field overstates the invoice, and the consequence of crossing a limit is not as certain as it looks.
The response is large (about 70 KB on a modest org: half `available_product_features`, a third `products[].tiers` and `products[].addons`), so make **one** call per run and pull only `billing_period`, `custom_limits_usd`, `discount_percent`, the four `projected_total_*` fields, `trial` / `free_trial_until`, and — for each product **and each entry in its `addons[]`**, since add-on meters like identified events, mobile recordings, and 30-day log retention carry their own tiers there — `type`, `usage_key`, `subscribed`, `tiers`, `free_allocation`, `current_usage`, `usage_limit`, `projected_usage`, and `projected_amount_usd`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Retain fixed credits when sizing materiality

When an organization has remaining account credits, this explicit projection drops discount_amount_usd, and the pricing reference adjusts the counterfactual only for discount_percent. Unlike the already-retained percentage discount, the supported fixed credit balance can fully absorb a $20–$500 step—_billing_with_credits.json contains a $500 example and BillingSummary.tsx:65-84 presents it as available credits—so the scout can report an immediate invoice increase when the actual effect is only credit burn; retain the field and apply it to both counterfactuals or frame the result as credit consumption.

AGENTS.md reference: products/signals/skills/AGENTS.md:L57-L57

Useful? React with 👍 / 👎.

@scheduled-actions-posthog

Copy link
Copy Markdown
Contributor

This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, please remove the stale label – otherwise this will be closed in another week. If you want to permanently keep it open, use the waiting label.

@scheduled-actions-posthog

Copy link
Copy Markdown
Contributor

This PR was closed due to lack of activity. Feel free to reopen if it's still relevant.

…ut-billing

# Conflicts:
#	products/signals/skills/AGENTS.md
@andrewm4894 andrewm4894 reopened this Nov 18, 2026
@andrewm4894 andrewm4894 removed the stale label Nov 18, 2026
@andrewm4894

Copy link
Copy Markdown
Member Author

@codex review

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 93a273b: 1 Must fix, 3 Should fix.

See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.

Comment on lines +208 to +209
- `pattern:billing:tiers:recording_count_in_period` — _"Subscribed; first tier is free to 15k/mo, marginal rate above that is $0.005/recording. Re-derive if the plan changes."_
- `pattern:billing:period-actual` — _"Period 2026-07-01 to 2026-07-31 billed $4,210 after discount and limits. Re-derive when `billing_period.current_period_start` moves past 2026-08-01."_

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 · Protect cached billing details with billing authorization

When the scout runs as a billing owner, these entries copy tier prices and the organization's invoice total into project-wide memory. SignalScratchpadViewSet.list checks project access and signal_scout:read, without billing authorization, so members denied billing access can retrieve those details. Keep this state behind billing authorization.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid. Memory no longer holds tier prices, invoice totals, or bill shares; the run re-reads them from the billing tools each time (a729283).

- **Materiality floor.** The move's projected impact on this period's invoice must clear **the greater of $20 and 2% of the last complete period's invoice** — so a small org still hears about small dollars and a large one is not paged over rounding.
The last period's invoice comes from summing `billing-spend-get` over the previous `billing_period` window, or from a cached `pattern:billing:period-actual`. When neither is available yet, use the $20 floor alone and say so in the memory entry.

**Group alias meters into families before testing concentration.** Several usage types are views or subsets of the same underlying usage, so counting them independently lets one real move put two series outside the tolerance and misread a localized billing change as a broad traffic story. Collapse each family to its primary meter and score the family once:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · Score paid add-ons independently before grouping for concentration

When identified-event usage or paid log retention jumps while total events or ingested MB stays flat, scoring each family once on its primary meter discards the anomaly. These add-ons have independent prices, so an unchanged primary meter does not imply unchanged cost. Score each paid meter independently and collapse families only for concentration.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid. Every meter is scored on its own; a family only counts once for the concentration test (a729283).

Comment on lines +99 to +100
1. **Discount.** Apply `discount_percent`. At 100% the impact is $0 no matter how large the usage move — real, and a legitimate reason not to report.
2. **Spending cap.** If the product's `usage_limit` (its spending limit expressed in units — see below) already binds, the incremental usage is not billed at all. Cap the impact at the remaining headroom under that limit, which can be $0.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · Apply the spending cap to both counterfactual prices

When both forecasts exceed a spending cap but current spend remains below it, capping the difference at remaining headroom overstates impact. With $100 accrued, a $1,000 cap, and forecasts of $1,100 and $1,500, this yields $400 despite both invoices being $1,000. Cap each counterfactual charge before subtracting, then apply the discount.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid. The cap now applies to each scenario before subtracting, and the discount after (a729283).

Distinguish an outage from batch lag with **project-scoped** evidence, because the overview's `usage_summary` is an organization total and a sibling project still emitting keeps it advancing while this project is dark:

- Zeros on one meter while this project's other meters report on the same date is **not** lag. Lag empties the whole day, not one series.
- When every meter on this project reads zero, re-run the usage call for that date with `breakdowns: ["type", "team"]` and no `team_ids`. Another project carrying data for the same date means ingestion is current, so this project's zeros are real. Only treat the day as lagging when no project has data for it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · Bound the organization-wide lag probe

When a project goes dark in a large organization, this unbounded type × team probe requests every meter on every project. The billing endpoint rejects oversized breakdowns with billing_query_too_large, leaving the scout without the evidence needed to distinguish an outage from batch lag. Use a project-only breakdown with top_projects: 1 to bound this existence check.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid. The probe now asks for the one date with top_projects: 1, so the other projects fold into one series (a729283).

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 93a273b99a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".


Apply the gates cheapest-first; most candidates die on the second:

1. **Machine-shaped?** Other meter families held? Then it is concentrated: carry on. If they moved too, run the **people check** once for the whole co-moving group, with `execute-sql` on this project:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Check traffic growth for single-family steps too

For an established project using only Product analytics, the other families are all zero and excluded from comparison, so an ordinary acquisition surge that increases both events and people 5× is automatically considered concentrated. This branch then skips the people check, allowing the step to pass deviation and dollar materiality and generate an instrumentation-anomaly report even though usage per person is unchanged. Run the people check for concentrated event steps too, at least when no other active family provides independent evidence that traffic held.

AGENTS.md reference: products/signals/skills/AGENTS.md:L65-L65

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid. The people check now also runs when no other family clears the floor to act as a comparison (ad6be3e).

- **Scan memory in two passes**, because `scout-scratchpad-search` defaults to 20 rows and this scout keeps a baseline per meter plus cursor, noise, dedupe, report, and reviewer entries — one project can exceed the default and silently drop the dedupe pointers you needed.
First a wide scan: `text=billing`, `keys_only=true`, `limit=200`. Then re-read only the candidate keys with an exact `key=` lookup. If the scan hits your limit, page back with `date_to` set to the oldest entry's `updated_at`.
- `scout-runs-list` with **`skill_name=signals-scout-billing`** and a 7-day bound. The fleet runs daily, so an unfiltered call spends its 20-row default on sibling scouts and your own last run drops out of the response.
- `inbox-reports-list` (`ordering=-updated_at`, `search`= the usage type) — a live report on a meter you are about to score is an **edit**, not a new report.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Search human-readable topics when checking existing reports

When a product scout has already reported the underlying problem without mentioning the billing service's raw usage identifier, this lookup cannot find it: SignalReportViewSet._apply_signal_report_search_filter searches only literal substrings of title and summary (products/signals/backend/views.py:2595–2599). For example, searching logs_and_traces_mb_in_period misses a logs scout's service-volume report, causing a separate billing report instead of the append_note required by the bundled reference. Search the human-readable product or meter name, or use semantic report search when available, and inspect the matching window before authoring.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid. The lookup now searches the meter's plain name and reads matches from the step's window (ad6be3e).

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed a729283: 1 Should fix.

Comment on lines +225 to +226
Lead with the money: what the bill does if this holds to the end of the period.
Evidence carries the usage type, current vs baseline volume, and the dollar delta this move is worth.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · Keep billing deltas behind billing authorization

When a scout runs as an owner in an organization with owner-only billing, this instruction publishes dollar deltas to members denied billing access. SignalReportViewSet requires inbox access without checking HasBillingUsageSpendReadAccess; removing absolute totals does not protect the remaining financial data. Gate billing report content by billing authorization.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid, and accepted for now. The scout registers disabled, so a person chooses to turn it on, and a report carries only the dollar delta of one move, never totals, prices or plan detail. Gating report visibility by billing access is a code change for a later PR.

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed ad6be3e: 1 Should fix.

This is review round 3 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.

GROUP BY day ORDER BY day
```

Compare `persons` on the stepped days against the same-weekday median before the step. **People up roughly in proportion to usage** is traffic growth or onboarding: write memory and stop. **People within ±20% while usage stepped** means each person now generates several times the usage they did, which no amount of growth does: treat what moved as one candidate, price each paid meter in it, and sum the impact for the materiality gate. When the project sends no events at all (a logs-only or warehouse-only project), the check cannot run: carry the group forward only if the step is at least 3× and has held for 2 or more complete days.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · Preserve going-dark findings in the no-events fallback

On a logs-only project, a paid meter falling to zero now reaches this fallback because no other family clears the volume floor. A zero latest value fails the at least 3× step requirement, so the scout suppresses the going-dark finding explicitly allowed in volume-only mode. Keep zero-usage outages eligible when the people check is unavailable.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid. A meter going dark now skips the people check, which only applies to usage that rose (113c960).

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 113c960: 3 Should fix.

This is review round 4 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.


For each usage type on this project, score **every complete day after your cursor**, oldest first, each against the **median of its own weekday over the trailing 4 weeks**.
On the weekly cadence that is about seven days, and one day when a run follows a run. After a run failed, was skipped, or the schedule slipped, it is the whole gap — and scoring only the newest bucket retires every day in between unscored, so a one-day spike or a meter that went dark and came back inside the gap is missed permanently.
Cap the catch-up at **10 days**: past that, score the 10 most recent and record the unscored span in `pattern:billing:cursor` rather than paging back through a month of series. Dates held in `pattern:billing:lag` rejoin this set once they fill.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · Extend the history window to cover the full catch-up baseline

After a scheduling gap of ten days, the oldest catch-up day needs a baseline observation from 38 days ago. The billing-usage-get call requests only ~35 days, leaving the oldest three days without the four same-weekday values required by the scoring rule. Fetch history starting 28 days before the earliest day being scored.

Comment on lines +37 to +39
HAVING events_yesterday + events_same_day_last_week > 0
AND (
weekly_run_due

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · Let the weekly backstop run without project events

When the designated project has no events on either sampled day, HAVING removes the row even when weekly_run_due is true. The harness skips the entire run, so sibling projects can drive spending-limit or trial-expiry warnings without the org-wide trajectory lane ever checking them. Make the weekly backstop independent of project event activity.

SELECT
countIf(timestamp >= toStartOfDay({now}) - INTERVAL 1 DAY) AS events_yesterday,
countIf(timestamp < toStartOfDay({now}) - INTERVAL 7 DAY) AS events_same_day_last_week,
{since} < {now} - INTERVAL 156 HOUR AS weekly_run_due

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · Preserve a retry path after a failed weekly run

With steady traffic, a failed weekly run still advances {since} because precheck_since includes failed run rows. This gate then suppresses another attempt for a week. The last successful run is now 14 days old, exceeding the 10-day catch-up cap and permanently dropping four days of coverage. Let failures bypass the cooldown or anchor it to successful runs.

@andrewm4894

Copy link
Copy Markdown
Member Author

Closing in favour of a fresh start: this work will be redone from a new branch through the PostHog inbox. This branch stays as prior art, including the review findings worked through here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant