Repository navigation
Add community skill: Funnel conversion scout - #19
Conversation
Template scout that watches one start-to-finish journey (entry event -> completion event) and reports a meaningful drop in mature 7-day entrant-cohort conversion, after ruling out capture breaks, identity changes, low volume, and seasonality. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: 2416c22a-dc98-49e2-944d-5be2786e8a73
|
@codex review |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fbb7b42061
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| Before you report a drop, check the things that produce the same shape without any change in user behaviour: | ||
|
|
||
| - **Capture.** Count both events per day for the last 30 days alongside the project's total daily events. If the completion event fell sharply on one date while other traffic held, that is a tracking break: a renamed event, a removed call, or a release that stopped sending it. Break the completion event down by `$lib`, `$lib_version`, or `$app_version` to find the release. |
There was a problem hiding this comment.
Verify a tracking failure before ruling out a conversion regression
When a real product failure prevents users from completing the journey—for example, a checkout release breaks payment submission—the completion event falls sharply while entry events and other traffic remain steady. This check categorically labels that exact signal a tracking break, and the later disqualifiers prevent reporting it as a conversion regression. The scout would therefore tell users their tracking or charts are wrong instead of identifying the broken journey. Require independent evidence of an instrumentation change before ruling out a conversion failure; falling completion-event volume alone cannot distinguish the two.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 6d457ab. A completion event that stops while entrants hold is now treated as a broken journey unless there is independent evidence of a tracking change: a renamed event starting the same day, other success signals that held, or the event vanishing from one release while users still reach success. Without that evidence the scout reports a conversion drop and names the tracking break as the alternative to rule out.
| ### Decide | ||
|
|
||
| - **Report** when the drop clears the bar and survives the measurement checks. Lead with what changed for users and how many fewer people completed, then the evidence, then the most likely cause and what to check next. | ||
| - **Edit** an open report on this journey with `scout-edit-report` instead of filing a new one while the drop persists. When conversion recovers into the baseline range, edit the report to say so. |
There was a problem hiding this comment.
Preserve the pre-drop baseline when checking for recovery
For a sustained regression, the rolling baseline eventually includes the degraded cohorts themselves. For example, after conversion drops from a stable 50% to a stable 20%, one week after the first fully affected cohort is scored, that 20% cohort becomes baseline cohort 1. The unchanged 20% current rate is then inside the baseline range and no longer below every baseline cohort, so this recovery instruction can mark the report recovered even though nothing improved. Save the pre-drop baseline with the open finding and use it to assess persistence and recovery rather than treating incorporation into the rolling baseline as recovery.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 6d457ab. The dedupe entry now saves the pre-drop baseline (median, range and cohort dates) when a report is filed. While the report is open, persistence and recovery are judged against that baseline, and cohorts after the onset are left out. Recovery means cohort 0 is back inside the pre-drop range. If the report is closed while conversion is still low, the rolling baseline takes over.
…re-drop baseline A completion event that stops while entrants hold is now treated as a broken journey unless there is independent evidence of a tracking change. Open reports are judged against the saved pre-drop baseline, so degraded cohorts rolling into the baseline no longer read as recovery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Generated-By: PostHog Desktop Task-Id: 2416c22a-dc98-49e2-944d-5be2786e8a73
Adds
skills/funnel-conversion-scout/, a template scout (kind: scout) that watches one product journey from the event that starts it to the event that completes it. It files an inbox report only when conversion drops meaningfully and the drop holds up after the measurement checks.Why: Many teams write this same "did fewer people finish the flow?" scout on their own, each with different event names, and the store has no template for it.
How it works
flow_name,entry_event,completion_event(required), plusconversion_window_days(default 7) andminimum_entries(default 50).scout-edit-reportwhile the drop lasts.query-funnel.For the maintainer to decide
conversion_window_daysbecause maturity depends on it. A daily-puzzle flow and an onboarding flow need very different windows.author_handleis left empty. Set it if you want attribution.Validation
python scripts/build_registry.pybuilds 10 entries, including this one askind: scoutwith itsscout_configand variables.registry.jsonwas restored and is not part of this PR.python scripts/safety_scan.pypasses.Created with PostHog Desktop from an inbox report
🤖 Generated with Claude Code
Origin