Skip to content

Add community skill: Funnel conversion scout - #19

Merged
andrewm4894 merged 2 commits into
mainfrom
posthog/funnel-conversion-scout
Nov 19, 2026
Merged

andrewm4894 merged 2 commits into
mainfrom
posthog/funnel-conversion-scout

Conversation

@posthog

@posthog posthog Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Adds skills/funnel-conversion-scout/, a template scout (kind: scout) that watches one product journey from the event that starts it to the event that completes it. It files an inbox report only when conversion drops meaningfully and the drop holds up after the measurement checks.

Why: Many teams write this same "did fewer people finish the flow?" scout on their own, each with different event names, and the store has no template for it.

How it works

  • Install variables: flow_name, entry_event, completion_event (required), plus conversion_window_days (default 7) and minimum_entries (default 50).
  • Runs one HogQL query to get nine rolling 7-day entrant cohorts. Each person counts once per cohort, and a cohort is scored only after its conversion window has fully closed. The query was checked against real events, and the SQL in the skill is identical to the version that ran.
  • Bar for a report: the drop must be at least 5 points and 20% of the baseline median, larger than 2× the binomial sampling noise, and below every baseline cohort. A drop that clears the bar by less than 2× has to hold on a run at least 3 days later before it is reported.
  • Before reporting, it checks capture breaks, identity changes, low volume, seasonality, and mix shift. One report per journey, updated with scout-edit-report while the drop lasts.
  • Checks that the install values have the right shape before using them in SQL. If an event name contains a quote or backslash, it switches to query-funnel.

For the maintainer to decide

  • Threshold. The candidate draft used a fixed 15-point drop. This version uses a relative-plus-noise bar instead. A fixed 15 points can never fire on a flow converting below 15%, and it is too loose for high-volume flows. The noise rule still requires about 14 points on a 50-entrant cohort at 50% conversion.
  • Added variable. I added conversion_window_days because maturity depends on it. A daily-puzzle flow and an onboarding flow need very different windows.
  • author_handle is left empty. Set it if you want attribution.

Validation

  • python scripts/build_registry.py builds 10 entries, including this one as kind: scout with its scout_config and variables. registry.json was restored and is not part of this PR.
  • python scripts/safety_scan.py passes.

Created with PostHog Desktop from an inbox report

🤖 Generated with Claude Code

Origin

  • Scout: a custom scout
  • First signal: 2026-11-17
  • Inbox report: open
  • Task started by: a person, from the inbox

Template scout that watches one start-to-finish journey (entry event -> completion event) and reports a meaningful drop in mature 7-day entrant-cohort conversion, after ruling out capture breaks, identity changes, low volume, and seasonality.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Generated-By: PostHog Desktop
Task-Id: 2416c22a-dc98-49e2-944d-5be2786e8a73
@posthog posthog Bot added the self-driving label Nov 17, 2026
@posthog
posthog Bot marked this pull request as ready for review November 17, 2026 20:02
@posthog
posthog Bot requested a review from andrewm4894 November 17, 2026 20:02
@andrewm4894

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-17T20:19:49.933339Z fbb7b42 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fbb7b42061

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread skills/funnel-conversion-scout/SKILL.md Outdated

Before you report a drop, check the things that produce the same shape without any change in user behaviour:

- **Capture.** Count both events per day for the last 30 days alongside the project's total daily events. If the completion event fell sharply on one date while other traffic held, that is a tracking break: a renamed event, a removed call, or a release that stopped sending it. Break the completion event down by `$lib`, `$lib_version`, or `$app_version` to find the release.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Verify a tracking failure before ruling out a conversion regression

When a real product failure prevents users from completing the journey—for example, a checkout release breaks payment submission—the completion event falls sharply while entry events and other traffic remain steady. This check categorically labels that exact signal a tracking break, and the later disqualifiers prevent reporting it as a conversion regression. The scout would therefore tell users their tracking or charts are wrong instead of identifying the broken journey. Require independent evidence of an instrumentation change before ruling out a conversion failure; falling completion-event volume alone cannot distinguish the two.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6d457ab. A completion event that stops while entrants hold is now treated as a broken journey unless there is independent evidence of a tracking change: a renamed event starting the same day, other success signals that held, or the event vanishing from one release while users still reach success. Without that evidence the scout reports a conversion drop and names the tracking break as the alternative to rule out.

Comment thread skills/funnel-conversion-scout/SKILL.md Outdated
### Decide

- **Report** when the drop clears the bar and survives the measurement checks. Lead with what changed for users and how many fewer people completed, then the evidence, then the most likely cause and what to check next.
- **Edit** an open report on this journey with `scout-edit-report` instead of filing a new one while the drop persists. When conversion recovers into the baseline range, edit the report to say so.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve the pre-drop baseline when checking for recovery

For a sustained regression, the rolling baseline eventually includes the degraded cohorts themselves. For example, after conversion drops from a stable 50% to a stable 20%, one week after the first fully affected cohort is scored, that 20% cohort becomes baseline cohort 1. The unchanged 20% current rate is then inside the baseline range and no longer below every baseline cohort, so this recovery instruction can mark the report recovered even though nothing improved. Save the pre-drop baseline with the open finding and use it to assess persistence and recovery rather than treating incorporation into the rolling baseline as recovery.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6d457ab. The dedupe entry now saves the pre-drop baseline (median, range and cohort dates) when a report is filed. While the report is open, persistence and recovery are judged against that baseline, and cohorts after the onset are left out. Recovery means cohort 0 is back inside the pre-drop range. If the report is closed while conversion is still low, the rolling baseline takes over.

…re-drop baseline

A completion event that stops while entrants hold is now treated as a broken journey unless there is independent evidence of a tracking change. Open reports are judged against the saved pre-drop baseline, so degraded cohorts rolling into the baseline no longer read as recovery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Generated-By: PostHog Desktop
Task-Id: 2416c22a-dc98-49e2-944d-5be2786e8a73
@andrewm4894
andrewm4894 merged commit 21ed732 into main Nov 19, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant