Repository navigation
feat(signals): add canonical billing scout - #86516
andrewm4894 wants to merge 16 commits into
Conversation
Adds `signals-scout-billing`, a canonical Signals scout that watches an organization's own PostHog bill through the customer-facing billing read tools (`billing-overview-get` / `billing-usage-get` / `billing-spend-get`). Two lanes per run: a step lane scoring each (usage type x project) pair's daily volume against its own same-weekday trailing baseline, and a trajectory lane reading projected period spend, spending limits, and tier position. Materiality is measured in dollars at the org's current tier position rather than in percent, and the body treats tier resets at the period boundary as the surface's defining false positive. Bundles `references/usage-types.md` for the usage-type to owning-scout routing table and the tier/period mechanics. Generated-By: PostHog Desktop Task-Id: e9d7c6c7-1798-49be-80e4-8ddcf8a7734c
|
🚫 This pull request was removed from the merge queue because it was pushed to by @andrewm4894. Please re-submit it in order to merge. See more details here.
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
|
Hey @andrewm4894! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
🤖 CI report
|
|
Reviews (1): Last reviewed commit: "feat(signals): add canonical billing sco..." | Re-trigger Greptile |
Addresses review feedback, each item verified against the implementation. Field semantics: - Derive the free threshold from the tier schedule, not `free_allocation`. A subscribed product can ship `free_allocation: 0` with its first tier free, so the old instruction classified wholly free usage as billable. - Quote a `*_with_limit*` projection when forecasting the invoice. The uncapped field is a hypothetical the customer is never billed. - Describe crossing a spending limit as a possibility. `never_drop_data` changes the outcome and the MCP tool strips it from the response. - Check the legacy `free_trial_until` alongside the `trial` object. - Branch on `breakdown_type` rather than assuming a two-element `breakdown_value`; type-only series carry one element and team ids come back as strings. - Price materiality at the current marginal tier and keep `billing-spend-get` inside one period, so a tier reset cannot clear the gate on arithmetic alone. - Store billing period boundaries with the cached period actual and re-derive once the period advances. Scoring and scope: - State the band numerically: robust z >= 3.5, a 50% fallback when MAD is zero, a 100-units/day volume floor, +/-20% concentration tolerance, and a materiality floor of the greater of $20 and 2% of last period's invoice. - Pin the step lane to this project's `team_ids` so multiple enabled projects in one org cannot file duplicate reports; note that the trajectory lane stays org-wide and release belongs on one project per organization. - Page the scratchpad with a `keys_only` scan and exact key re-reads rather than relying on the 20-row default. - Define an evidence-backed reviewer ladder and file unrouted rather than guess, since no runtime source names the billing owner. Also reformats the markdown with oxfmt to fix the failing frontend formatting check. Generated-By: PostHog Desktop Task-Id: e9d7c6c7-1798-49be-80e4-8ddcf8a7734c
|
Worked through all 11 Codex findings. Verified each against the implementation first — every one held up, so all are addressed except the P1 rollout gate, which cannot be a code change (replied on that thread: Field semantics that were wrong and would have produced false or overstated reports:
Scoring and scope:
Period rollover also fixed: the cached period actual now stores its The failing check was markdown formatting — both files reformatted with the catalog-pinned oxfmt 0.57.0; Still unverified and unchanged from the original description: none of this has run against live billing data. |
Dogfooded against the billing read tools on a real organization: type-only breakdowns return a string breakdown_value, today's bucket reads zero rather than partial, spend keys series by product rather than usage type, and usage_limit is the spending limit in units.
Second review pass, layered on top of the dogfooding commit. Most of these are consequences of the previous round's fixes rather than of the original body. Would have broken every run: - Pass `breakdowns` / `team_ids` as JSON-encoded strings. The MCP handler types all three array params as Zod strings, so an actual array is rejected before the request is made. Self-contradictions introduced by the last review round: - The usage-only grant was told to run the step lane, which then required overview tier data for its mandatory dollar gate. It now runs in volume-only mode, reporting just the two shapes decisive without a dollar figure - a meter going dark, and a >=5x step holding three or more days - and carrying the rest as memory. - The usage cursor close-out stopped the whole run, so a sparse designated project could suppress an org-wide spending-limit or trial-expiry warning. It now gates lane 1 only. - The volume floor tested the baseline median alone, discarding a meter rising from near zero to a large paid spike - the cold-start instrumentation loop the scout most wants to catch. It now skips only when both the baseline and the latest value are under the floor, and a zero baseline is handled as a new-meter onset. Correctness: - Price a delta as `price(with) - price(without)` across the tier schedule, then discount and cap. A flat rate times the delta overstates any move that crosses a tier, and a fully discounted or already-capped meter can add $0 while clearing the gate. Keeps the observed per-tier `current_usage` / `projected_usage` detail for finding the marginal tier, and caps against the observed units-denominated `usage_limit`. - Group alias and subset meters into families before the concentration test. `group_analytics` is a view of identified-event usage and the logs retention tier is the same content, so counting them separately let one real move read as a broad traffic story. - Route `ai_credits_used_in_period` away from the AI-observability scout. Those credits are metered from PostHog's own regional projects, not the customer's `$ai_*` events, so that scout would close out finding nothing. The dogfooded vocabulary table agrees: they bill under the `posthog_ai` product, not `llm_analytics`. - Point the materiality gate at that vocabulary table, since tiers and limits are keyed by product rather than by usage type. - Pass `skill_name` to `scout-runs-list`; the fleet's daily schedules push this scout's own last run outside the 20-row default. Generated-By: PostHog Desktop Task-Id: e9d7c6c7-1798-49be-80e4-8ddcf8a7734c
…ling data Fallback order for pricing a step when the overview is denied, truncated, or lacks tiers; treat every trajectory field as optional; use the flat materiality floor when no prior-period invoice exists; calibrate on the first runs and report unknown response shapes through agent-feedback.
|
Round two addressed, and rebuilt on top of the dogfooding commit rather than over it — One would have broken every run. Three were self-contradictions the previous round introduced, which is the useful part of a second pass:
Pricing was wrong in a way that could clear the gate at $0. Two routing corrections. Alias meters (
CI green. Rollout gates unchanged and still outside this diff: add to |
… persistence pricing Score a complete all-zero day instead of skipping to the last non-zero one; keep addons, discount_percent, and free_allocation from the overview; price a step as it projects to period end; count simultaneous meter onsets in the concentration gate; forecast post-trial charges only for active autosubscribe trials.
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 1 · PR risk: 0/10 Reviews |
…od math Keep `projected_usage` in the overview pull list so the materiality counterfactual has its starting unit count. Distinguish batch lag from a real project outage with project-scoped evidence rather than the org-wide usage summary. Group the PostHog Desktop aggregate meter with its token and compute components before the concentration test. Normalize period-over-period spend to a daily run rate so a longer calendar month is not read as a trajectory finding. Tell reports to carry only the billing figures the finding needs, since the inbox is team-scoped. Generated-By: PostHog Desktop Task-Id: 4a683539-86ee-427e-ac90-b9178a7fcfa6
|
/trunk merge |
|
Gates denied: this PR touches billing logic (deny-listed) and is sized above the auto-approvable tier; it also has multiple unresolved reviewer findings (a billing-data-exposure concern and several P2 correctness issues in the pricing/materiality math) that haven't all been confirmed fixed on the current head.
Gate mechanics and policy version
|
|
This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, please remove the |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Four fixes from the latest review round on the billing scout: - Lane 1 now scores every complete day after the cursor, oldest first and capped at 7, so a spike inside a run gap is not retired unscored. Consecutive days of one step stay one report. - The concentration test only compares against families that clear the volume floor, so a 1 -> 0 wobble no longer rejects a material candidate. - Delta pricing anchors both scenarios to the service's own projected_usage instead of adding the step on top of a forecast that already contains it, which slid both endpoints up the tier curve and invented impact across a free threshold. - A step's impact is described as what it adds to this period, not as being above last period; period-over-period stays with the trajectory lane. Generated-By: PostHog Desktop Task-Id: 96734564-6ffa-4a67-aaa2-5dcfd10b2e1e
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fcc51f06f7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| - Projected period spend materially above the last complete period's actual. | ||
| **Attribute it, do not require it to be unattributable.** A gradual single-product ramp is the shape lane 1 structurally cannot see — a trailing same-weekday baseline climbs with the ramp, so no day is ever an outlier — and it is exactly what this lane exists for. When one product explains the higher forecast, name that product in the report; only skip when lane 1 already filed the same meter this period. | ||
| **Compare daily run rates, not raw totals.** Billing periods are calendar months of different lengths, so a March projection sits ~11% above an unchanged February at an identical daily rate — enough to clear the materiality floor on the calendar alone. Divide each side by its own period's day count, or scale the prior total to the current period's length, before calling the difference surprising. |
There was a problem hiding this comment.
Exclude flat charges from period-length normalization
When an invoice includes a monthly flat charge, dividing the whole total by each period's day count creates a trajectory change even when the charge is unchanged; for example, a fixed $1,000 invoice appears about 10.7% more expensive per day in February than January. Fresh evidence beyond the earlier period-length fix is the supported nonzero flat_amount_usd in frontend/src/mocks/fixtures/_billing_with_flat_fee.json:464, which the pricing reference explicitly includes, so normalize only usage-proportional charges and compare fixed charges directly.
Useful? React with 👍 / 👎.
| The response is `results`: a list of series, each carrying `label`, `dates`, `data`, `breakdown_type`, and `breakdown_value`, plus a top-level `team_id_options` listing every project the caller can see. | ||
|
|
||
| **`breakdown_value`'s shape follows `breakdown_type` — branch on it, never index blindly.** | ||
| With `["type"]`, `breakdown_type` is `"type"` and `breakdown_value` is a **plain string** (`"recording_count_in_period"`). |
There was a problem hiding this comment.
Accept singleton arrays for type breakdowns
When billing-usage-get returns the supported singleton-array shape for a type breakdown, this branch expects a string and cannot match the usage type to the vocabulary or tier schedule. The Billing UI explicitly normalizes both string and array values in frontend/src/scenes/billing/billingUsageLogic.ts:42-53,74, and the production-shaped billing test uses ["billable_feature_flag_requests_count_in_period"] with breakdown_type="type"; unwrap a singleton array as well or valid meters fall into the unknown, volume-only path.
AGENTS.md reference: products/signals/skills/AGENTS.md:L57-L57
Useful? React with 👍 / 👎.
| - A caller who cannot see a sibling project has no way to tell the two apart. Score the zero day and say in the report that a delayed batch has not been ruled out, rather than dismissing it. | ||
|
|
||
| Use the org `usage_summary` only to corroborate, never to decide. | ||
| When the day really is lagging, score the last non-zero day but **leave the cursor at the last day you actually scored**, and record the lagging dates separately as `pattern:billing:lag`. |
There was a problem hiding this comment.
Avoid rescoring the cursor date during lag
Fresh evidence after the cursor fix is that, during an ordinary multi-day batch delay, the last nonzero day is usually the date already stored in pattern:billing:cursor, so this fallback scores the same bucket on every scheduled run while leaving the cursor unchanged. If that bucket belongs to a live step episode, line 229 can append the same evidence and running total repeatedly even though edit_report(append_note=...) is explicitly non-idempotent in products/signals/backend/scout_harness/prompt.py:404-407; score the fallback only when it is strictly newer than the cursor, otherwise skip lane 1 and continue the trajectory lane.
AGENTS.md reference: products/signals/skills/AGENTS.md:L57-L57
Useful? React with 👍 / 👎.
| ### Lane 2 — the trajectory (every run, one call) | ||
|
|
||
| `billing-overview-get` carries the forecast. Read the field mechanics in [`references/usage-types.md`](references/usage-types.md) before quoting a number — the wrong projection field overstates the invoice, and the consequence of crossing a limit is not as certain as it looks. | ||
| The response is large (about 70 KB on a modest org: half `available_product_features`, a third `products[].tiers` and `products[].addons`), so make **one** call per run and pull only `billing_period`, `custom_limits_usd`, `discount_percent`, the four `projected_total_*` fields, `trial` / `free_trial_until`, and — for each product **and each entry in its `addons[]`**, since add-on meters like identified events, mobile recordings, and 30-day log retention carry their own tiers there — `type`, `usage_key`, `subscribed`, `tiers`, `free_allocation`, `current_usage`, `usage_limit`, `projected_usage`, and `projected_amount_usd`. |
There was a problem hiding this comment.
Retain fixed credits when sizing materiality
When an organization has remaining account credits, this explicit projection drops discount_amount_usd, and the pricing reference adjusts the counterfactual only for discount_percent. Unlike the already-retained percentage discount, the supported fixed credit balance can fully absorb a $20–$500 step—_billing_with_credits.json contains a $500 example and BillingSummary.tsx:65-84 presents it as available credits—so the scout can report an immediate invoice increase when the actual effect is only credit burn; retain the field and apply it to both counterfactuals or frame the result as credit consumption.
AGENTS.md reference: products/signals/skills/AGENTS.md:L57-L57
Useful? React with 👍 / 👎.
|
This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, please remove the |
|
This PR was closed due to lack of activity. Feel free to reopen if it's still relevant. |
…ut-billing # Conflicts: # products/signals/skills/AGENTS.md
|
@codex review |
There was a problem hiding this comment.
PostHog Review · Standard · reviewed 93a273b: 1 Must fix, 3 Should fix.
See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.
| - `pattern:billing:tiers:recording_count_in_period` — _"Subscribed; first tier is free to 15k/mo, marginal rate above that is $0.005/recording. Re-derive if the plan changes."_ | ||
| - `pattern:billing:period-actual` — _"Period 2026-07-01 to 2026-07-31 billed $4,210 after discount and limits. Re-derive when `billing_period.current_period_start` moves past 2026-08-01."_ |
There was a problem hiding this comment.
P1 · Protect cached billing details with billing authorization
When the scout runs as a billing owner, these entries copy tier prices and the organization's invoice total into project-wide memory. SignalScratchpadViewSet.list checks project access and signal_scout:read, without billing authorization, so members denied billing access can retrieve those details. Keep this state behind billing authorization.
There was a problem hiding this comment.
Valid. Memory no longer holds tier prices, invoice totals, or bill shares; the run re-reads them from the billing tools each time (a729283).
| - **Materiality floor.** The move's projected impact on this period's invoice must clear **the greater of $20 and 2% of the last complete period's invoice** — so a small org still hears about small dollars and a large one is not paged over rounding. | ||
| The last period's invoice comes from summing `billing-spend-get` over the previous `billing_period` window, or from a cached `pattern:billing:period-actual`. When neither is available yet, use the $20 floor alone and say so in the memory entry. | ||
|
|
||
| **Group alias meters into families before testing concentration.** Several usage types are views or subsets of the same underlying usage, so counting them independently lets one real move put two series outside the tolerance and misread a localized billing change as a broad traffic story. Collapse each family to its primary meter and score the family once: |
There was a problem hiding this comment.
P2 · Score paid add-ons independently before grouping for concentration
When identified-event usage or paid log retention jumps while total events or ingested MB stays flat, scoring each family once on its primary meter discards the anomaly. These add-ons have independent prices, so an unchanged primary meter does not imply unchanged cost. Score each paid meter independently and collapse families only for concentration.
There was a problem hiding this comment.
Valid. Every meter is scored on its own; a family only counts once for the concentration test (a729283).
| 1. **Discount.** Apply `discount_percent`. At 100% the impact is $0 no matter how large the usage move — real, and a legitimate reason not to report. | ||
| 2. **Spending cap.** If the product's `usage_limit` (its spending limit expressed in units — see below) already binds, the incremental usage is not billed at all. Cap the impact at the remaining headroom under that limit, which can be $0. |
There was a problem hiding this comment.
P2 · Apply the spending cap to both counterfactual prices
When both forecasts exceed a spending cap but current spend remains below it, capping the difference at remaining headroom overstates impact. With $100 accrued, a $1,000 cap, and forecasts of $1,100 and $1,500, this yields $400 despite both invoices being $1,000. Cap each counterfactual charge before subtracting, then apply the discount.
There was a problem hiding this comment.
Valid. The cap now applies to each scenario before subtracting, and the discount after (a729283).
| Distinguish an outage from batch lag with **project-scoped** evidence, because the overview's `usage_summary` is an organization total and a sibling project still emitting keeps it advancing while this project is dark: | ||
|
|
||
| - Zeros on one meter while this project's other meters report on the same date is **not** lag. Lag empties the whole day, not one series. | ||
| - When every meter on this project reads zero, re-run the usage call for that date with `breakdowns: ["type", "team"]` and no `team_ids`. Another project carrying data for the same date means ingestion is current, so this project's zeros are real. Only treat the day as lagging when no project has data for it. |
There was a problem hiding this comment.
P2 · Bound the organization-wide lag probe
When a project goes dark in a large organization, this unbounded type × team probe requests every meter on every project. The billing endpoint rejects oversized breakdowns with billing_query_too_large, leaving the scout without the evidence needed to distinguish an outage from batch lag. Use a project-only breakdown with top_projects: 1 to bound this existence check.
There was a problem hiding this comment.
Valid. The probe now asks for the one date with top_projects: 1, so the other projects fold into one series (a729283).
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 93a273b99a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| Apply the gates cheapest-first; most candidates die on the second: | ||
|
|
||
| 1. **Machine-shaped?** Other meter families held? Then it is concentrated: carry on. If they moved too, run the **people check** once for the whole co-moving group, with `execute-sql` on this project: |
There was a problem hiding this comment.
Check traffic growth for single-family steps too
For an established project using only Product analytics, the other families are all zero and excluded from comparison, so an ordinary acquisition surge that increases both events and people 5× is automatically considered concentrated. This branch then skips the people check, allowing the step to pass deviation and dollar materiality and generate an instrumentation-anomaly report even though usage per person is unchanged. Run the people check for concentrated event steps too, at least when no other active family provides independent evidence that traffic held.
AGENTS.md reference: products/signals/skills/AGENTS.md:L65-L65
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Valid. The people check now also runs when no other family clears the floor to act as a comparison (ad6be3e).
| - **Scan memory in two passes**, because `scout-scratchpad-search` defaults to 20 rows and this scout keeps a baseline per meter plus cursor, noise, dedupe, report, and reviewer entries — one project can exceed the default and silently drop the dedupe pointers you needed. | ||
| First a wide scan: `text=billing`, `keys_only=true`, `limit=200`. Then re-read only the candidate keys with an exact `key=` lookup. If the scan hits your limit, page back with `date_to` set to the oldest entry's `updated_at`. | ||
| - `scout-runs-list` with **`skill_name=signals-scout-billing`** and a 7-day bound. The fleet runs daily, so an unfiltered call spends its 20-row default on sibling scouts and your own last run drops out of the response. | ||
| - `inbox-reports-list` (`ordering=-updated_at`, `search`= the usage type) — a live report on a meter you are about to score is an **edit**, not a new report. |
There was a problem hiding this comment.
Search human-readable topics when checking existing reports
When a product scout has already reported the underlying problem without mentioning the billing service's raw usage identifier, this lookup cannot find it: SignalReportViewSet._apply_signal_report_search_filter searches only literal substrings of title and summary (products/signals/backend/views.py:2595–2599). For example, searching logs_and_traces_mb_in_period misses a logs scout's service-volume report, causing a separate billing report instead of the append_note required by the bundled reference. Search the human-readable product or meter name, or use semantic report search when available, and inspect the matching window before authoring.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Valid. The lookup now searches the meter's plain name and reads matches from the step's window (ad6be3e).
…cing Unverified: docs
…name Unverified: docs
| Lead with the money: what the bill does if this holds to the end of the period. | ||
| Evidence carries the usage type, current vs baseline volume, and the dollar delta this move is worth. |
There was a problem hiding this comment.
P2 · Keep billing deltas behind billing authorization
When a scout runs as an owner in an organization with owner-only billing, this instruction publishes dollar deltas to members denied billing access. SignalReportViewSet requires inbox access without checking HasBillingUsageSpendReadAccess; removing absolute totals does not protect the remaining financial data. Gate billing report content by billing authorization.
There was a problem hiding this comment.
Valid, and accepted for now. The scout registers disabled, so a person chooses to turn it on, and a report carries only the dollar delta of one move, never totals, prices or plan detail. Gating report visibility by billing access is a code change for a later PR.
Unverified: docs
There was a problem hiding this comment.
PostHog Review · Standard · reviewed ad6be3e: 1 Should fix.
This is review round 3 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.
| GROUP BY day ORDER BY day | ||
| ``` | ||
|
|
||
| Compare `persons` on the stepped days against the same-weekday median before the step. **People up roughly in proportion to usage** is traffic growth or onboarding: write memory and stop. **People within ±20% while usage stepped** means each person now generates several times the usage they did, which no amount of growth does: treat what moved as one candidate, price each paid meter in it, and sum the impact for the materiality gate. When the project sends no events at all (a logs-only or warehouse-only project), the check cannot run: carry the group forward only if the step is at least 3× and has held for 2 or more complete days. |
There was a problem hiding this comment.
P2 · Preserve going-dark findings in the no-events fallback
On a logs-only project, a paid meter falling to zero now reaches this fallback because no other family clears the volume floor. A zero latest value fails the at least 3× step requirement, so the scout suppresses the going-dark finding explicitly allowed in volume-only mode. Keep zero-usage outages eligible when the people check is unavailable.
There was a problem hiding this comment.
Valid. A meter going dark now skips the people check, which only applies to usage that rose (113c960).
Unverified: docs
There was a problem hiding this comment.
PostHog Review · Standard · reviewed 113c960: 3 Should fix.
This is review round 4 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.
|
|
||
| For each usage type on this project, score **every complete day after your cursor**, oldest first, each against the **median of its own weekday over the trailing 4 weeks**. | ||
| On the weekly cadence that is about seven days, and one day when a run follows a run. After a run failed, was skipped, or the schedule slipped, it is the whole gap — and scoring only the newest bucket retires every day in between unscored, so a one-day spike or a meter that went dark and came back inside the gap is missed permanently. | ||
| Cap the catch-up at **10 days**: past that, score the 10 most recent and record the unscored span in `pattern:billing:cursor` rather than paging back through a month of series. Dates held in `pattern:billing:lag` rejoin this set once they fill. |
There was a problem hiding this comment.
P2 · Extend the history window to cover the full catch-up baseline
After a scheduling gap of ten days, the oldest catch-up day needs a baseline observation from 38 days ago. The billing-usage-get call requests only ~35 days, leaving the oldest three days without the four same-weekday values required by the scoring rule. Fetch history starting 28 days before the earliest day being scored.
| HAVING events_yesterday + events_same_day_last_week > 0 | ||
| AND ( | ||
| weekly_run_due |
There was a problem hiding this comment.
P2 · Let the weekly backstop run without project events
When the designated project has no events on either sampled day, HAVING removes the row even when weekly_run_due is true. The harness skips the entire run, so sibling projects can drive spending-limit or trial-expiry warnings without the org-wide trajectory lane ever checking them. Make the weekly backstop independent of project event activity.
| SELECT | ||
| countIf(timestamp >= toStartOfDay({now}) - INTERVAL 1 DAY) AS events_yesterday, | ||
| countIf(timestamp < toStartOfDay({now}) - INTERVAL 7 DAY) AS events_same_day_last_week, | ||
| {since} < {now} - INTERVAL 156 HOUR AS weekly_run_due |
There was a problem hiding this comment.
P2 · Preserve a retry path after a failed weekly run
With steady traffic, a failed weekly run still advances {since} because precheck_since includes failed run rows. This gate then suppresses another attempt for a week. The last successful run is now 14 days old, exceeding the 10-day catch-up cap and permanently dropping four days of coverage. Let failures bypass the cooldown or anchor it to successful runs.
|
Closing in favour of a fresh start: this work will be redone from a new branch through the PostHog inbox. This branch stays as prior art, including the review findings worked through here. |

Problem
An organization learns that its PostHog bill jumped when the invoice arrives. No scout watches the organization's own bill (origin thread, recent prompt).
Changes
signals-scout-billing, files an inbox report when the bill moves and names the product scout that owns the cause.Note
The scout seeds disabled on new projects under the fleet's current seed allowlist. A person turns it on.
A report shows the dollar delta of one move to everyone who can read the project inbox, including members without billing access. Accepted for now.
How did you test this code?
Test rationale: None added. The diff is a skill body with no code path.
Release status
Docs update
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Agent: PostHog Slack app, Claude Opus 5; Claude Code, Fable 5.2
Skills:
/authoring-scouts,/writing-pr-descriptionsSecurity review: needed
More context for AI agents
pricing_reset_dates, which can differ from the renewal date.SKILL.md, the "Machine-shaped?" gate and the memory rule. Security: reports put billing deltas in a team-scoped inbox.Created with PostHog from a Slack thread