Skip to content

fix(signals): stop repository selection treating app events as vendor work - #111226

Closed
posthog[bot] wants to merge 3 commits into
masterfrom
posthog-self-driving/fixsignals-stop-repository-selection-4cdf18
Closed

posthog[bot] wants to merge 3 commits into
masterfrom
posthog-self-driving/fixsignals-stop-repository-selection-4cdf18

Conversation

@posthog

@posthog posthog Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Problem

  • Inbox users get reports routed to the wrong repository, or to no repository, when the project's own events have vendor-like names.
  • The selection prompt said to return null for "billing" or "internal ops" questions, with no ownership check. A project's own subscription_* event read as a vendor billing question.
  • The research prompt told every team's agent that it was investigating "the PostHog codebase", which invites the same mix-up.
  • Research can clone and verify a different repository, but nothing fed that answer back. The report kept the selection agent's earlier guess, and reviewers and the PR targeted it.

Origin

  • Scout: a custom scout
  • First signal: 2026-10-03
  • Inbox report: open
  • Task started by: auto-start, after the report was rated P3 and ready to fix

Changes

  • Ownership rule in selection: the agent must find explicit evidence (behavior inside the vendor's own service, and no candidate with the emitting code) before it calls the work vendor-owned. A billing or infrastructure word in an event name is no longer a null case. reason must name the ownership evidence.
  • Ownership rule in research: the preamble now names "this project's codebase" and says the project's data describes the application it instruments.
  • Research reconciles the selection: the presentation turn can return code_repository. When research found the code elsewhere, the activity replaces the selection before it resolves reviewers and persists artefacts.
Research names Selection agent's guess becomes
A connected repository That repository, autostart_eligible=False
A repository the installation cannot reach Cleared (repository=None), so the next run selects again
Nothing, or the same repository Unchanged
  • Only a selection agent's guess (it carries a task id) yields. A pin, the single connected repository, a correction by a person or a scout, and a selection research already wrote all stay.
  • An empty connected-repository list means the installation is unavailable, so the guess stays rather than being cleared.
  • The existing mid-run supersede guard still applies, so a reviewer correction written during the run still wins.

Note

A reconciled selection never autostarts a PR, the same as a reviewer's wrong-repo correction. A person starts the run from the report.

How did you test this code?

  • Ran products/tasks/backend/tests/test_repo_selection_prompt.py, products/signals/backend/test/test_agentic_report_activity.py, and products/signals/backend/test/test_research_prompt.py locally against the dev stack database.
  • Ran ruff check, ruff format, and hogli lint:tach on the changed files.
  • Not checked: a live sandbox run of selection or research. The LLM effect of the prompt change is not measured here.

Test rationale:

  • test_run_agentic_report_activity_reconciles_the_selection_with_research covers the regression where research finds the code in another repository and the report keeps the earlier guess. The two replace and clear cases fail with the reconcile step removed. The closest existing tests (..._keeps_a_scout_repository_correction, ..._keeps_reviewer_selection_written_mid_run) only cover corrections written by others.
  • test_prompt_asks_for_ownership_evidence_before_a_vendor_verdict guards the ownership section and its place before the null rules, next to the existing prompt-section tests.

Release status

  • No feature flag controls this change
  • This change is behind a feature flag and is not available to users
  • This change makes a previously flagged feature available to everyone

Automatic notifications

  • Publish to changelog?

Docs update

  • Updated products/signals/backend/report_generation/AGENTS.md to describe the reconcile step.

🤖 Agent context

Autonomy: Fully autonomous

Agent: PostHog Desktop (Claude Code), claude-opus-5-5

  • Started from an Inbox report about wrong repository targets for customer events. No customer material is in the diff. Test event and repository names are invented.
  • Skills invoked: /writing-pr-descriptions.
  • No duplicate: an open PR search for repository selection and research found #109624 (keeps report content when selection fails) and #102950 (scout pinned repositories). Neither covers ownership evidence or research reconciliation.
  • Rejected alternative: always clear a conflicting selection. A connected repository that research verified is better evidence than the guess, so it replaces the guess instead.

Created with PostHog Desktop from this inbox report.

🤖 Generated with Claude Code

… work

Repository selection and research now treat a project's own events, errors, and logs as its own application, and ask for explicit evidence before they call the behavior vendor-owned.

Research can name the repository that holds its code evidence. The research activity then replaces the selection agent's guess with that repository when it is connected, and clears the guess when it is not. Pins and corrections by people or scouts stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Generated-By: PostHog Desktop
Task-Id: e0cddaa9-2fac-4e82-8296-54df7caa2338
@posthog posthog Bot added the self-driving label Oct 3, 2026
@posthog
posthog Bot marked this pull request as ready for review October 3, 2026 04:53
@trunk-io

trunk-io Bot commented Oct 3, 2026

Copy link
Copy Markdown

Merging to master in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

@parameterai

parameterai Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Risk: No findings

This increment replaces the forgeable reason-prefix marker with a server-set from_research boolean on RepoSelectionResult: the flag is stamped only by Signals research when it rewrites the selection, forced to False on every selection-agent output, and stripped from the prompt schema, so the model can no longer fake a research-written selection and block later reconciliation. I found no new vulnerabilities in the increment.

Sentinel reviewed 71af1e0 · Review settings

@pr-assigner-resolver-posthog
pr-assigner-resolver-posthog Bot requested review from a team October 3, 2026 04:54
stamphog[bot]

This comment was marked as outdated.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

🤖 CI report

⚠️ Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

✅ Duplication (Python) — clean

New Python code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

✅ Duplication (TypeScript) — clean

New TypeScript code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

@hosthog

hosthog Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

HostHog preview — posthog-desktop-web

The previews for this PR have been torn down and no longer serve.

@coderabbitai

coderabbitai Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

🧰 Additional context used
📚 Code guidelines (8)
.agents/security.md — configured
.agents/skills/sending-notifications/SKILL.md — configured
docs/published/handbook/engineering/type-system.md — configured
.agents/skills/writing-tests/SKILL.md — configured
docs/internal/person-data-access.md — configured
.agents/skills/adopting-generated-api-types/SKILL.md — configured
.claude/commands/conventions.md — configured
.agents/skills/writing-code-comments/SKILL.md — configured

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: PostHog/posthog/.coderabbit.yaml
  • Review profile: QUIET
  • Plan: Enterprise
  • Run ID: 20dbcbbf-4f13-45db-8bb8-cd8a68fd57ad
📥 Commits

Reviewing files that changed from the base of the PR and between bdf55c5 and 71af1e0.

📒 Files selected for processing (4)
  • products/signals/backend/temporal/agentic/report.py
  • products/signals/backend/test/test_agentic_report_activity.py
  • products/tasks/backend/logic/repo_selection/agent.py
  • products/tasks/backend/logic/repo_selection/types.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 8 remain after this review.


📝 Walkthrough

Walkthrough

Report research can identify a code repository other than the initially selected repository. The report activity reconciles that repository with connected repositories, then persists and returns the updated selection. The repository-selection prompt distinguishes application ownership from vendor ownership and clarifies when to return a null result.

Priority: ⬇️ Low

Merge Risk: 🔵 Low · up to 71af1

A stale repository cache can temporarily clear a valid selection and delay report processing, but autostart is disabled and the repository can be rediscovered.

Security Architecture Review

Security architecture risk: 🟡 Moderate · up to 71af1

A repository correction can leave an older reviewer identity active for the new destination. Cached connection data can also incorrectly clear a valid destination. Project-level access checks and existing execution limits contain the exposure, but destination and authorization state are not consistently updated together.

Retained concerns

  • Medium · security · inferred: The new automatic repository replacement does not invalidate reviewers derived for the previous repository. Persistence updates reviewers only when they are nonempty and a finding changed, so a replacement can leave repository B paired with repository A’s machine-derived reviewer. Execution resolves that reviewer to a user without binding the identity to the selected repository; research’s autostart-ineligible flag does not block this reviewer path. Under otherwise satisfied execution gates, work may therefore start under an identity whose supporting evidence belongs to the old destination. The retention behavior existed before this PR, but research-driven retargeting adds a new exposure to it.
  • Low · architecture · observed: The new reconciliation treats absence from a nonempty cached repository list as confirmed installation inaccessibility and persists an empty selection. The provider deliberately accepts stale snapshots without refreshing them, so a currently connected repository omitted from that snapshot can clear the report’s destination. This is a state-ownership and recovery concern, not evidence of a GitHub authorization bypass: downstream work is blocked until a new selection is established.
Security review details

Security Blast Radius

  • inferred — The supported exposure is within a report’s project and its connected repository set, with execution potentially using an eligible organization member’s identity. The implementation handoff supplies that user, repository, and the signals_implementation scope preset. No cross-tenant access or broader credential authority was established.

Security Findings and Attack Paths

  • inferred — Content influencing research attribution could retarget an existing report while its old machine-derived reviewers remain live. If a retained reviewer resolves to a member whose autonomy threshold permits the report, execution can use that identity despite the replacement’s autostart-ineligible flag. This requires connected-repository membership and the remaining execution gates; attacker steering and end-to-end exploitation were not demonstrated.

Trust Boundaries and Controls

  • observed — autostart_eligible=False is not a universal execution veto: it prevents the reviewer-less fallback. Reviewer execution still requires member resolution and an autonomy threshold; user-edited reviewer execution runs as the editing user rather than a named colleague. The team disable switch applies across these paths.
  • observed — The replacement dispatcher does not directly start work merely because a decision was queued. Its worker claims a reservation and re-enters current-state execution evaluation, which loads the latest repository and eligibility. Before doing so, it calls an existing inferred-repository refresh whose guard checks eligibility but not the new research-provenance field; the setter’s full effects were not verified.

Resilience and Maintainability Implications

  • observed — Missing repository state blocks implementation evaluation. Existing execution controls also check current report content, replacement-decision eligibility, and concurrent task creation. These controls contain duplicate or obsolete execution, but do not establish repository-version consistency for retained reviewers.

Hardening Proposals

  • proposed — Bind machine-derived reviewers to a repository-selection version and invalidate or recompute them when that destination changes, including when resolution is empty. Enforce that binding before selecting an execution identity.
  • proposed — Represent membership freshness and unavailable states separately from confirmed denial before clearing a destination. Also distinguish research attribution from text-derived inference at refresh boundaries, rather than using execution eligibility as a provenance discriminator.
🚥 Pre-merge checks | ✅ 1
✅ Passed checks (1 passed)
Check name Status Explanation
Description check ✅ Passed The description explains the problem, user-visible changes, reconciliation rules, tests, release status, and agent context. It is mostly complete, though it omits the agent session link, CodeRabbit CL…
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments (1)
products/signals/backend/temporal/agentic/report.py-748-748 (1)

748-748: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Preserve the selection when repository connectivity is unavailable.

resolve_team_github_integration excludes unavailable integrations and returns None. list_team_connected_repositories converts that state to [], so the reconciliation branch treats an unavailable lookup as a confirmed non-member and clears the existing selection. Return a distinct unavailable state and preserve the selection for it. Continue clearing the selection only when repository membership was successfully checked and the repository is absent.


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Repository: PostHog/posthog/.coderabbit.yaml
  • Review profile: QUIET
  • Plan: Enterprise
  • Run ID: 116e18b0-f968-4faa-a8e5-16fad47666fb
📥 Commits

Reviewing files that changed from the base of the PR and between cf416e9 and 2f084b6.

📒 Files selected for processing (6)
  • products/signals/backend/report_generation/AGENTS.md
  • products/signals/backend/report_generation/research.py
  • products/signals/backend/temporal/agentic/report.py
  • products/signals/backend/test/test_agentic_report_activity.py
  • products/tasks/backend/logic/repo_selection/agent.py
  • products/tasks/backend/tests/test_repo_selection_prompt.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

…n reconcile

A selection that research already wrote now stays on later runs, because only a selection agent guess yields. An empty connected-repository list means the installation is unavailable, so the guess stays instead of being cleared. The selection prompt now counts a match in a candidate that holds the vendor's own service as evidence for that candidate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Generated-By: PostHog Desktop
Task-Id: e0cddaa9-2fac-4e82-8296-54df7caa2338
@stamphog
stamphog Bot dismissed their stale review October 3, 2026 05:11

A new stamphog review started for this PR — the fresh verdict replaces this approval.

@posthog

posthog Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Re the minor note on unavailable connectivity: fixed in bdf55c5. An empty connected-repository list now keeps the selection agent's guess, because the selection agent only ran with candidates, so an empty list means the installation is unavailable. A new test case covers it.

stamphog[bot]

This comment was marked as outdated.

@trunk-io

trunk-io Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Static Badge   Static Badge   Static Badge

Failed Test Failure Summary Logs
test_moves_only_desktop_canvas_comments_and_mirrors_and_reverses A database migration attempted to add an index concurrently inside a transaction, which is not supported. Logs ↗︎

View Full Report ↗︎ ⋅ Docs

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments (2)
products/signals/backend/temporal/agentic/report.py-752-754 (1)

752-754: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Do not clear a repository guess based on a stale cache.

The selection agent refreshes the cache when needed, but reconciliation later reads it with allow_refresh=False. If that nonempty snapshot omits the repository found by research, reconciliation can clear the selection, disable autostart, and say GitHub cannot reach the repository. Use a fresh membership check before clearing; preserve the selection when freshness is unknown.

products/signals/backend/temporal/agentic/report.py-745-751 (1)

745-751: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Track selection provenance outside reason.

The selection agent supplies reason as model output, but the guard treats any reason with the research prefix as proof that research wrote the selection. If the agent returns that prefix and research finds a different repository, _reconcile_repo_selection can return the guess unchanged with autostart_eligible=True. Add an explicit provenance field to RepoSelectionResult and skip reconciliation only for a research-produced selection.


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Repository: PostHog/posthog/.coderabbit.yaml
  • Review profile: QUIET
  • Plan: Enterprise
  • Run ID: 253686c1-e5cd-4e31-a4d8-d7b789f3d070
📥 Commits

Reviewing files that changed from the base of the PR and between 2f084b6 and bdf55c5.

📒 Files selected for processing (3)
  • products/signals/backend/temporal/agentic/report.py
  • products/signals/backend/test/test_agentic_report_activity.py
  • products/tasks/backend/logic/repo_selection/agent.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.

The reconcile guard read the selection reason prefix, which is model output and can be forged. A new `from_research` field on `RepoSelectionResult` now marks a research write. It is stripped from the selection prompt schema and reset after the selection agent runs, so the model cannot set it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Generated-By: PostHog Desktop
Task-Id: e0cddaa9-2fac-4e82-8296-54df7caa2338
@stamphog
stamphog Bot dismissed their stale review October 3, 2026 05:29

A new stamphog review started for this PR — the fresh verdict replaces this approval.

@posthog

posthog Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

CodeRabbit notes: the provenance note is fixed in 71af1e0 (from_research field). Not changed: the stale-cache note. Reconcile reads the light cache without a refresh, the same way list_team_connected_repositories does for its other callers, so that unattended runs do not trigger GitHub refreshes. In the worst case a stale cache clears a guess, and the report then asks for a repository and blocks autostart. It never targets a wrong repository. A reviewer can decide whether a fresh check is worth the extra GitHub call.

stamphog[bot]

This comment was marked as outdated.

@scheduled-actions-posthog

Copy link
Copy Markdown
Contributor

This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, please remove the stale label – otherwise this will be closed in another week. If you want to permanently keep it open, use the waiting label.

@scheduled-actions-posthog

Copy link
Copy Markdown
Contributor

This PR was closed due to lack of activity. Feel free to reopen if it's still relevant.

@github-actions github-actions Bot added the feature/agent-infra Feature Tag: Agent infra label Oct 20, 2026
@jakesciotto

Copy link
Copy Markdown
Contributor

Reopening: the third fix for a customer whose inbox reports land on the wrong repository. Two earlier fixes shipped (#99705, #105789), and this one closes the remaining gap.

@jakesciotto jakesciotto reopened this Nov 3, 2026
@stamphog
stamphog Bot dismissed their stale review November 3, 2026 00:36

A new stamphog review started for this PR — the fresh verdict replaces this approval.

@posthog
posthog Bot requested a review from andrewm4894 November 3, 2026 00:36

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not approved yet — waiting on the conditions below.

Re-add the stamphog label to request another review once you have addressed this.

The prerequisites gate refused this pull request because it has merge conflicts. The other gates passed: no deny-list categories matched, and the size and tier checks were within limits. Because the conflicts need to be resolved first, stamphog can't proceed with it. To move forward, the author can resolve the conflicts by rebasing or merging the base branch and pushing the update, or ask a human reviewer to take a look.

Gate mechanics and policy version
Gate Result
prerequisites ✗ merge conflicts present
deny-list ✓ no deny categories matched
size ✓ 115L, 4F substantive, 205L/7F incl. docs/generated/snapshots — within ceiling
tier ✓ T1-agent / T1c-medium (205L, 7F, single-area, fix)
stamphog 2.7.0 .stamphog/policy.yml @ unknown · reviewed head 71af1e0

@jakesciotto jakesciotto removed the stale label Nov 3, 2026

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 71af1e0: 1 Must fix, 1 Should fix.

See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.

Comment on lines +761 to +762
autostart_eligible=False,
from_research=True,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 · Block reviewer-based autostart after a research correction

When a corrected report has a reviewer who meets the autonomy threshold, maybe_autostart_implementation_task selects that reviewer regardless of repository_autostart_eligible; this flag only guards the reviewer-less fallback. The report therefore opens a PR in the research-selected repository without a person starting it. Gate automatic starts on from_research too.

Comment on lines 1077 to +1081
await _persist_agentic_report_artefacts(
input.team_id,
input.report_id,
result,
input.repo_selection,
repo_selection,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · Refresh reviewers when research changes the repository

When re-research changes repositories while confirming all existing findings, _persist_agentic_report_artefacts skips the new reviewer list because has_new_finding is false. The report retains the old repository's reviewers even after resolving the new owners, misrouting review and autostart identity. Refresh pipeline-selected reviewers whenever the repository changes.

@scheduled-actions-posthog

Copy link
Copy Markdown
Contributor

This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, please remove the stale label – otherwise this will be closed in another week. If you want to permanently keep it open, use the waiting label.

@scheduled-actions-posthog

Copy link
Copy Markdown
Contributor

This PR was closed due to lack of activity. Feel free to reopen if it's still relevant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants