Skip to content

feat(ssf): add protocol endpoints and stream provisioning - #132007

Draft
ReeceJones wants to merge 1 commit into
codex/ssf-servicefrom
codex/ssf-protocol
Draft

ReeceJones wants to merge 1 commit into
codex/ssf-servicefrom
codex/ssf-protocol

Conversation

@ReeceJones

@ReeceJones ReeceJones commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Reviewers need smaller diffs to assess the SSF implementation in #123731.

Changes

Expose discovery, keys, streams, verification, polling and receipt endpoints. Add upstream stream provisioning and retry handling.

Part 3 of seven. Previous: #132006. Next: #132008. Parts 1–4 form the backend stack; parts 5–7 form a second stack based on #132008.

How did you test this code?

The combined tree exactly matches #123731 at 98c422d. Commit hooks and whitespace checks passed. Runtime tests and visual verification remain pending.

Test rationale: Preserve the original tests and assertions; no new coverage accompanies this mechanical split.

Release status

  • No feature flag controls this change

Docs update

No documentation behavior changed during this split.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Codex, GPT-6. Original implementation by ReeceJones; the agent redistributed unchanged files.

Skills: /stacking-prs, /writing-pr-descriptions, /improving-drf-endpoints, /implementing-mcp-tools, /routing-outbound-api-calls, /writing-code-comments, /verifying-changes.

The original base is retained. Rebase conflicts with current master and full validation remain outstanding before review readiness.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

🤖 CI report

⚠️ Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

✅ Duplication (Python) — clean

New Python code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

✅ Duplication (TypeScript) — clean

New TypeScript code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

❌ Django migration risk — blocked migration detected

We've analyzed your migrations for potential risks.

Summary: 2 Safe | 0 Needs Review | 1 Blocked

❌ Blocked

Causes locks or breaks compatibility

posthog.1412_ssf_connections
  │  └─ #1 ⚠️ AddField
  │     Adding NOT NULL field with callable default (list) - verify it's stable
  │     model: ssfstream, field: audiences, default: list
  │  └─ #2 ✅ AddField
  │     Adding NOT NULL field with constant default (safe in PG11+)
  │     model: ssfstream, field: endpoint_url
  │  └─ #3 ⚠️ AddField
  │     Adding NOT NULL field with callable default (list) - verify it's stable
  │     model: ssfstream, field: events_delivered, default: list
  │  └─ #4 ✅ AddField
  │     Adding NOT NULL field with constant default (safe in PG11+)
  │     model: ssfstream, field: last_error
  │  └─ #5 ✅ AddField
  │     Adding nullable field requires brief lock
  │     model: ssfstream, field: last_verified_at
  │  └─ #6 ✅ AddField
  │     Adding nullable field requires brief lock
  │     model: ssfstream, field: operation_until
  │  └─ #7 ✅ AddField
  │     Adding nullable field requires brief lock
  │     model: ssfstream, field: receiver_token
  │  └─ #8 ✅ AddField
  │     Adding NOT NULL field with constant default (safe in PG11+)
  │     model: ssfstream, field: receiver_token_hash
  │  └─ #9 ✅ AddField
  │     Adding NOT NULL field with constant default (safe in PG11+)
  │     model: ssfstream, field: remote_stream_id
  │  └─ #10 ✅ AddField
  │     Adding nullable field requires brief lock
  │     model: ssfstream, field: retry_at
  │  └─ #11 ✅ AddField
  │     Adding nullable field requires brief lock
  │     model: ssfstream, field: verification_requested_at
  │  └─ #12 ✅ AddField
  │     Adding nullable field requires brief lock
  │     model: ssfstream, field: verification_state
  │  └─ #13 ⚠️ AlterField
  │     Field alteration may cause table locks or data loss (check if changing type or constraints)
  │     model: ssfevent, field: stream, field_type: ForeignKey
  │  └─ #14 ⚠️ AlterField
  │     Field alteration may cause table locks or data loss (check if changing type or constraints)
  │     model: ssfstream, field: status, field_type: CharField
  │  └─ #15 ✅ CreateModel
  │     Creating new table is safe
  │     model: SSFConnection
  │  └─ #16 ✅ AddField
  │     Adding nullable field requires brief lock
  │     model: ssfstream, field: connection
  │  └─ #17 ⚠️ AddConstraint
  │     Adding constraint may lock table (use NOT VALID pattern)
  │     model: ssfstream
  │  └─ #18 ⚠️ RunPython: RunPython data migration needs review for performance
  │
  └──> ℹ️  INFO:
       ℹ️  Skipped operations on newly created tables (empty tables
       don't cause lock contention).
  │
  └──> �[91m⚠️  COMBINATION RISKS:�[0m
       ❌ BLOCKED: #18 RunPython + #1 AddField, #2 AddField, #3 AddField,
       #4 AddField, #5 AddField, #6 AddField, #7 AddField, #8 AddField,
       #9 AddField, #10 AddField, #11 AddField, #12 AddField, #13
       AlterField, #14 AlterField, #15 CreateModel, #16 AddField, #17
       AddConstraint    RunPython data migration combined with schema
       changes. Data migrations can hold locks during execution,
       especially on large tables. Split into separate migrations: 1)
       schema changes, 2) data migration.

✅ Safe

Brief or no lock, backwards compatible

posthog.1411_ssf
  └─ #1 ✅ CreateModel
     Creating new table is safe
     model: SSFConfiguration
  └─ #2 ✅ CreateModel
     Creating new table is safe
     model: SSFStream
  └─ #3 ✅ CreateModel
     Creating new table is safe
     model: SSFEvent
  │
  └──> ℹ️  INFO:
       ℹ️  Skipped operations on newly created tables (empty tables
       don't cause lock contention).
posthog.1413_ssf_connection_transmitter_url
  └─ #1 ✅ AddField
     Adding NOT NULL field with constant default (safe in PG11+)
     model: ssfconnection, field: transmitter_url

📚 How to Deploy These Changes Safely

AddConstraint:

Add constraints in 2 phases without locking, using the PostHog helpers:

  1. AddConstraintNotValid (instant, validates new rows only, no table scan)

  2. ValidateConstraint in a separate migration (scans table with non-blocking lock)

    from posthog.migration_helpers import AddConstraintNotValid, ValidateConstraint

See the migration safety guide

AddField:

This operation acquires a brief lock but doesn't rewrite the table.

Deployment uses lock timeouts with automatic retries, so lock contention will cause retries rather than connection pile-up.

RunPython:

Use batching for large data migrations:

  • Use .iterator() to avoid loading all rows into memory
  • Use .bulk_update() instead of saving individual objects
  • Batch size: 1,000-10,000 rows per batch
  • Add pauses between batches
  • Consider background jobs for very large updates (millions of rows)

See the migration safety guide

Last updated: 2026-11-09 17:14 UTC (0df4be1)

@scheduled-actions-posthog

Copy link
Copy Markdown
Contributor

This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, please remove the stale label – otherwise this will be closed in another week. If you want to permanently keep it open, use the waiting label.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant