Skip to content

perf(triangulation): keep API responsive during overlap work - #673

Draft
frgfm wants to merge 1 commit into
mainfrom
codex/extract-geospatial-inference
Draft

frgfm wants to merge 1 commit into
mainfrom
codex/extract-geospatial-inference

Conversation

@frgfm

@frgfm frgfm commented Aug 4, 2026 •

Copy link
Copy Markdown
Member

Summary

Triangulation blocks the backend's event loop, so unrelated requests wait for the calculation. Run the same calculation in one bounded worker thread per backend process. Keep the current algorithm, results, dependencies, and deployment.

This replaces the original service extraction with a smaller latency fix: 5 files, +9 net production lines, +45 net lines including tests. Both alert attachment and alert refresh use the same worker limit.

flowchart LR
    C["Requests"] --> A
    subgraph B["Existing backend process"]
        A["API event loop"] -->|"await"| T["One geometry worker thread"]
        T -->|"same result"| A
    end
    A --> D["Database and storage"]
Loading

Measured results and decision

Same optimized algorithm and package versions, one backend worker, five fresh runs per option:

Option Peak stack RSS (MiB) Health p95 under load (ms) 80-view calculation request (ms)
Current main 181.6 1,437.3 379.1
Bounded thread — this PR 186.9 26.6 380.4
Optimized HTTP split — prototype 243.6 17.1 390.7

RSS is resident process memory. The memory column includes backend and compute processes. The p95 is the response time within which 95% of health requests finish. Values are medians of each run's peak memory, p95 health time, and median calculation time.

The thread cuts unrelated-request tail latency by 98.1%, with 2.9% more RSS. Calculation time stays roughly unchanged. The split adds 34.1% RSS at one backend worker, so this PR adds no service, HTTP call, token, network, or dependency.

At four backend workers, a separate two-run experiment saved 12.8% RSS and 8.6% proportionally shared memory with the split. That is a different deployment from the repository's one-worker configuration. Native libraries remain in the backend in this PR; process and database isolation are deferred.

Raw samples, pinned environment, benchmark code, and reproduction instructions.

Conditions: native Linux ARM64, CPython 3.11.17, one CPU quota shared by the whole stack. Each mixed-load condition sends the same 12 geometry requests and 96 scheduled health requests. All geometry and timezone output hashes match across all 21 conditions.

Limits: these are local synthetic HTTP diagnostics with real backend imports. Database, S3, notification delivery, and background jobs are excluded. The split is an optimistic loopback prototype. These are not VPS ingestion measurements or Docker image-size savings.

Test plan

  • Full backend suite against migrated PostgreSQL and S3 test services: 684 passed, 1 skipped.
  • Regression test verifies worker-thread execution, a responsive event loop, serialized geometry jobs, and unchanged results.
  • Backend Docker build and startup; make quality; make precommit.

@codecov

codecov Bot commented Aug 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.57%. Comparing base (9f4c75f) to head (c7fd372).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #673      +/-   ##
==========================================
+ Coverage   94.48%   94.57%   +0.09%     
==========================================
  Files          59       59              
  Lines        3246     3246              
==========================================
+ Hits         3067     3070       +3     
+ Misses        179      176       -3     
Flag Coverage Δ
backend 94.72% <100.00%> (+0.09%) ⬆️
client 91.30% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@frgfm

frgfm commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

Adversarial review completed across OpenCode, Claude, Codex, and Cursor.\n\nApplied:\n- deploy workflow uploads the updated Compose file and provisions the shared inference token\n- CPU triangulation runs off the event loop and remains serialized per process\n- backend exposes a stable 503 response without leaking upstream details\n- zero-angle cone compatibility and concurrency/health regression coverage\n\nKept intentionally:\n- notification timezone lookup failures skip delivery, matching the agreed best-effort policy\n- no Redis, local fallback, or shared geospatial abstraction without evidence they are needed

@frgfm frgfm self-assigned this Aug 4, 2026
@frgfm
frgfm force-pushed the codex/extract-geospatial-inference branch from 6d997b7 to c7fd372 Compare October 15, 2026 13:19
@frgfm frgfm changed the title feat(inference): extract geospatial workloads from backend perf(triangulation): keep API responsive during overlap work Oct 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant