Skip to content

feat: add in-memory sensor statistics with batched SQLite flush - #179

Open
kage-zero wants to merge 3 commits into
honeynet:mainfrom
kage-zero:feat/sensor-statistics-9
Open

kage-zero wants to merge 3 commits into
honeynet:mainfrom
kage-zero:feat/sensor-statistics-9

Conversation

@kage-zero

Copy link
Copy Markdown
Contributor

Adds server-side sensor statistics using write-behind caching. Event counts are aggregated in memory and periodically flushed to SQLite in a single transaction, so there is no database write per event on POST /api/publish.

Closes #9

Changes:

  • Added a StatsCollector that keeps per-sensor totals and unflushed deltas in Go maps, protected by a sync.RWMutex.
  • POST /api/publish now calls StatsCollector.Increment(sensorID) and returns 202 Accepted without taking a SQLite write lock.
  • Added a sensor_stats table (sensor_id, event_count, last_event_at), created on startup if it doesn't exist.
  • Added a background worker that flushes deltas to SQLite every 5-10s. It swaps the delta map under a short lock, then upserts all modified sensors in one transaction.
  • Added GET /api/sensors/stats, which returns the logged-in user's sensor counts straight from RAM (0 DB queries).
  • Added throttled WebSocket broadcasts (every 500ms-1s) so /mysensors updates counts live without updating on every raw event.
  • Saved counts are loaded from SQLite into memory on startup.
  • If a flush fails (e.g. SQLite is busy), the unflushed deltas are merged back and retried on the next cycle, so no counts are lost.
  • On SIGINT/SIGTERM, the worker stops cleanly and does a final synchronous flush before the process exits.

How it works:

Sensor event -> POST /api/publish -> StatsCollector (RAM)

  • GET /api/sensors/stats reads from RAM
  • WebSocket broadcast (batched, 500ms-1s)
  • Background worker (5-10s) -> SQLite sensor_stats

Testing:

  • Publish events and confirm counts update on /mysensors in real time
  • Confirm GET /api/sensors/stats returns only the authenticated user's sensors
  • Restart the backend and confirm counts persist
  • Stop the server with SIGINT/SIGTERM and confirm pending deltas are flushed
  • Verify there are no SQLite writes per event under load

Notes:

For a single-instance deployment, an in-memory map keeps things lightweight with no external dependencies. If the backend later scales horizontally, the map can be replaced with Redis (HINCRBY/INCRBY, plus Pub/Sub for WebSocket fan-out).

@kage-zero

Copy link
Copy Markdown
Contributor Author

Hi @glaslos , just a gentle ping on this PR. Let me know if anything needs changing. Thanks!

@glaslos

glaslos commented Oct 11, 2026

Copy link
Copy Markdown
Member

AI usage disclosure required?

Comment thread backend/entities/sensor.go Resolved
Comment thread backend/entities/sensor.go Resolved
Comment thread backend/handlers/publish.go Resolved
Comment thread backend/repos/sensors.go Resolved
Comment thread backend/server.go Resolved
Comment thread backend/websocket.go Resolved
@kage-zero

Copy link
Copy Markdown
Contributor Author

AI usage disclosure required?

Yes, I used AI for code generation. However, I independently researched the problem, designed the architecture, identified the files and functions required, and determined the overall implementation approach. The code was generated with AI assistance, and I personally tested the implementation using both black-box and white-box testing.

I'm also going through the issues you've highlighted in the review and working on the necessary changes. I'll push the updates to the existing PR shortly for your review.

I'm happy to disclose my use of AI and take responsibility for the final contribution.

@kage-zero

Copy link
Copy Markdown
Contributor Author

Thanks for the review and suggestions! I have refactored the sensor statistics implementation to address all feedback.

Summary of Changes

  1. Created StatsService (backend/services/stats.go):

    • Moved all in-memory statistics state (sensor event totals, timestamps, unflushed deltas, dirty flags for WebSocket broadcast, mutexes, and retrieval logic) out of SensorRepo into a dedicated StatsService.
    • SensorRepo now strictly handles SQLite persistence (GetAllStats and BatchUpsertStats).
  2. Moved Flusher & WebSocket Broadcaster to the Service:

    • The periodic flush loop (5s) and WebSocket broadcast loop (500ms) are now owned by StatsService.
    • Persistence is transactional in SQLite; if a flush fails, pending deltas are preserved in-memory so counts are not lost and can be retried.
    • Lifecycle is managed cleanly via Start(ctx) and Stop(ctx), ensuring a final flush to SQLite during graceful application shutdown.
  3. Moved WebSocket Handler to Handlers Package:

    • Moved StatsSubscribe into backend/handlers/sensors.go and wired /api/sensors/stats/ws via backend/routes.go.
    • Cleaned up statistics-specific logic from backend/websocket.go.
  4. Always Use Backend UTC Timestamp:

    • Updated backend/handlers/publish.go to always record time.Now().UTC().Format(time.RFC3339) for last_event_at.
    • Untrusted sensor-supplied timestamps in incoming payloads are ignored for statistics.
  5. Entity Tags & Column Mapping Convention:

    • Removed redundant db struct tags from Sensor and SensorStat in backend/entities/sensor.go to match the project's "json" mapper convention.
  6. Standardized EventCount to int64:

    • Changed counter type from uint64 to int64 for native compatibility with SQLite's signed 64-bit INTEGER and Go's SQL driver scanning.
    • Added table constraint CHECK(event_count >= 0) to the SQLite schema.
    • Added in-memory guards capping values at math.MaxInt64 to prevent integer overflow.

Verification & Tests

  • Service Unit Tests (backend/services/stats_test.go):
    • TestStatsService_IncrementAndQuery: in-memory counting and retrieval.
    • TestStatsService_FlushAndRevertOnError: batch upsert and delta retention on DB error.
    • TestStatsService_LifecycleStartStop: clean start, cancel, and final flush.
    • TestStatsService_WebSocketBroadcastAndSubscribe: live WebSocket broadcast updates.
    • TestStatsService_BoundaryAndOverflowProtection: verified boundary checks and math.MaxInt64 cap.
  • Repository Tests (backend/repos/sensors_test.go):
    • TestSensorRepo_StatsBoundaryAndConstraint: verified persistence of signed 64-bit max values and rejection of negative counts.
    • TestSensorRepo_StatsPersistence: verified transactional batch upserts.
  • Handler Tests (backend/handlers/publish_test.go):
    • TestPublish_AlwaysUsesCurrentUTCTime: verified that incoming sensor timestamps are ignored in favor of the backend UTC clock.
  • Integration Tests (backend/server_test.go):
    • TestSensorStats_RESTEndpointAndIsolation (PASS)
    • TestSensorStats_FlushAndReload (PASS)
    • TestSensorStats_WebSocketBroadcast (PASS)
  • Frontend Checks:
    • npm run check (0 errors, 0 warnings)
    • npm test (35 passed, 35 total)
  • Formatting:
    • Formatted all Go files with gofmt -s -w.

AI Assistance Disclosure

In accordance with CONTRIBUTING.md:

  • Tool used: Google Antigravity.
  • Scope of use: Drafting the architecture refactoring (extracting StatsService, wiring lifecycle into server.go), adapting unit/integration tests for int64 SQLite compatibility, and drafting this review response.
  • Human verification: All code, SQL queries, package interfaces, and tests were inspected line-by-line, verified against the repository standards, and executed locally.

@kage-zero
kage-zero force-pushed the feat/sensor-statistics-9 branch from 02946fd to 9fef003 Compare October 12, 2026 11:37
@kage-zero
kage-zero requested a review from glaslos October 12, 2026 13:33
@kage-zero

Copy link
Copy Markdown
Contributor Author

Hi @glaslos ,
I have re-synced the branch with main and resolved the merge conflicts across backend/handlers, backend/repos, and backend/server:

  • Preserved the new sensor UUID authentication check (RequireRegisteredSensors) alongside the stats increments.
  • Integrated the stats service lifecycle into the graceful server shutdown and database closing flow.
  • All Go and Svelte CI checks are passing green.
    Ready for review when you have a moment. Thanks!

Comment thread backend/repos/sensors.go
Comment on lines +204 to +216
for id, delta := range deltas {
if delta <= 0 {
continue
}
stat := entities.SensorStat{
SensorID: id,
EventCount: delta,
LastEventAt: lastSeen[id],
}
if _, err := stmt.Exec(stat); err != nil {
return err
}
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you make this a single statment instead of a loop?

Comment thread backend/services/stats.go Outdated
Comment on lines +62 to +70
// SetFlushInterval overrides the default periodic flush interval.
func (s *StatsService) SetFlushInterval(d time.Duration) {
s.flushInterval = d
}

// SetBroadcastInterval overrides the default broadcast interval.
func (s *StatsService) SetBroadcastInterval(d time.Duration) {
s.broadcastInterval = d
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an anti pattern, pass the config to the constructor

Comment thread backend/services/stats.go Outdated

// Start loads initial stats from the database and launches periodic background tasks.
func (s *StatsService) Start(ctx context.Context) error {
if s.repo != nil {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happens if repo is nil?

Comment thread backend/services/stats.go

// Stop stops background workers and commits any pending deltas to the database.
func (s *StatsService) Stop() error {
if s.cancel != nil {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when is cancel nil?

Comment thread backend/services/stats.go Outdated
if s.cancel != nil {
s.cancel()
}
s.wg.Wait()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how long does this wait?

Comment thread backend/services/stats.go Outdated
}

// GetStatsForSensors retrieves current stats for the requested sensor IDs from memory.
// It matches by prefix when an event used a truncated 8-character ID or full UUID.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this an issue?

Comment thread backend/services/stats.go Outdated
for {
select {
case msg := <-sub.msgs:
ctxTimeout, cancel := context.WithTimeout(ctx, 5*time.Second)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shorten to 1s

Comment thread backend/services/stats.go Outdated
}

// PublishStats sends a message to all active WebSocket stats subscribers.
func (s *StatsService) PublishStats(msg []byte) bool {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unused return

Comment thread backend/services/stats.go Outdated
case <-ctx.Done():
return
case <-ticker.C:
_ = s.Flush()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what if this errors?

@kage-zero
kage-zero force-pushed the feat/sensor-statistics-9 branch from 97a9cc0 to 64cd4c4 Compare October 28, 2026 05:02

@glaslos glaslos left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes, mainly for the unauthenticated stats WebSocket. Details inline.

Comment thread backend/routes.go Outdated
r.Post("/login", h.Login)
// Shared event links are unguessable IDs and must work without login.
r.Get("/events/{id}", h.GetEventByID)
r.Get("/sensors/stats/ws", h.StatsSubscribe)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This route is outside the auth group and broadcasts stats for every sensor of every user, including the full sensor UUIDs. Those UUIDs are what guards publish and /download, so anyone can harvest them. Please move it behind auth and filter the broadcast by owner.

Comment thread backend/services/stats.go Outdated
}
if count == 0 {
for totalID, c := range s.totals {
if (strings.HasPrefix(totalID, id) || strings.HasPrefix(id, totalID)) && c > 0 {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The prefix fallback lets another sensor's counts be attributed to the requested one. Publishing with an ID equal to the first 8 chars of a victim's UUID inflates their total. Match on the exact ID only.

Comment thread backend/handlers/sensors.go Resolved
kage-zero and others added 2 commits November 5, 2026 00:04
…ynet#9)

Add server-side sensor statistics using write-behind caching. Event counts are aggregated in memory and periodically flushed to SQLite in a single transaction, avoiding database writes per event on POST /api/publish.

- Add StatsService managing in-memory sensor event totals, flusher, and WebSocket broadcaster lifecycle
- Add sensor_stats table (sensor_id, event_count, last_event_at) with batch upsert in SensorRepo
- Add GET /api/sensors/stats authenticated endpoint returning user's sensor counts directly from memory
- Add GET /api/sensors/stats/ws WebSocket endpoint for real-time stats updates
- Increment sensor stats on POST /api/publish using backend UTC timestamps
- Update SensorList.svelte to display live event counts with WebSocket updates
- Ensure non-negative counters and boundary protection against integer overflow
- Add unit and integration tests for service, repo, handler, and server lifecycle

Co-Authored-By: Google Antigravity <antigravity@google.com>
…t delete race

- Move /api/sensors/stats/ws behind authentication and filter live broadcasts by sensor owner
- Match sensor stats on exact ID only in GetStatsForSensors, removing prefix fallback
- Add tombstones for deleted sensors and hold lock during SQLite batch flush to avoid re-insert races
- Pass auth token parameter in SensorList.svelte when opening stats WebSocket
- Add tests for auth rejection, broadcast isolation, exact matching, and tombstone protection

Co-Authored-By: Google Antigravity <antigravity@google.com>
@kage-zero

Copy link
Copy Markdown
Contributor Author

Hey @glaslos, moved /api/sensors/stats/ws behind authentication with owner-filtered broadcasts, switched to exact ID matching in GetStatsForSensors, and added tombstoning/locking during flush to eliminate the delete race. All frontend and backend tests pass; ready for review!

@kage-zero
kage-zero force-pushed the feat/sensor-statistics-9 branch from 64cd4c4 to dc1a019 Compare November 4, 2026 18:57

@glaslos glaslos left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are still comments not addressed. Specifically in the stats service. I also want the Flush to not block the /api/publish endpoint

Comment thread backend/routes.go
r.Get("/", h.Index)
r.Get("/global.css", h.CSS)
r.Get("/myqueries", h.Index)
r.Get("/mysensors", h.Index)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This route is already registered

…ring flush

- Remove duplicate /mysensors static route in backend/routes.go
- Unblock /api/publish by executing SQLite flush outside s.mu lock
- Pass StatsConfig to NewStatsService constructor instead of setter methods
- Enforce non-nil repository on StatsService.Start
- Bound StatsService.Stop with timeout to prevent shutdown hangs
- Log background flush errors instead of silently discarding
- Shorten WebSocket write timeout from 5s to 1s
- Remove unused return on PublishStats
- Batch upsert stats in a single multi-row SQL statement instead of a loop
- Add unit test verifying IncrementStat is not blocked during SQLite flush

Co-Authored-By: Google Antigravity <antigravity@google.com>
@kage-zero

Copy link
Copy Markdown
Contributor Author

Hey @glaslos, Sorry, I missed a few things earlier. Anyway, I’ve addressed all the remaining feedback, everything is cleared now, and all tests are passing. The changes are ready for your review!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sensor server-side statistics

2 participants