Skip to content

fix: report a failed firehose stream to the caller of the handler - #19

Merged
Peeja merged 1 commit into
mainfrom
claude/firehose-error-observable
Sep 18, 2026
Merged

Peeja merged 1 commit into
mainfrom
claude/firehose-error-observable

Conversation

@Peeja

@Peeja Peeja commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

[From Claude:]

One line in firehoseHandler, plus tests. Independent of #18 — that one is client-side, this is the server handler.

The problem

firehoseHandler writes an error event and then return nil (app.go:321), so a stream that died on a store failure looks to echo exactly like one that reached the end of the data.

There is no logging or metrics machinery in the repo today to notice, so nothing is broken right now. The reason to change it now is that adding that machinery later would not pick these up. Echo's request logger derives what it reports from exactly two things (request_logger.go:353 onward, v4.15.0):

err := next(c)
...
v.Status = res.Status
...
if config.LogError && err != nil { v.Error = err }

This handler defeats both. It returns nil, and response.WriteHeader(http.StatusOK) (app.go:313) commits the status before the first record, so the status is fixed before anything can fail.

I measured what standard middleware sees today, with RequestLoggerWithConfig{LogStatus, LogError, HandleError} in front of the real handler and a store that fails mid-stream:

HTTP status on the wire : 200
request logger status   : 200
request logger error    : <nil>
response body           : "event: error\ndata: {\"error\":\"getting revocations: connection refused\"}\n\n"

The database is down, the consumer is told, and the request logs as a clean 200. Same two inputs feed RED metrics, so it would count as a success. The signal is erased in the handler, below the layer that would go looking for it — so this is not something the observability work can fix on its own later.

The change

Return the error instead of nil. Same setup, after:

request logger status   : 200
request logger error    : streaming revocations: getting revocations: connection refused
body still intact       : true

It costs nothing today. With no middleware registered, echo routes the error to DefaultHTTPErrorHandler, which returns immediately on a committed response (echo.go:428) — nothing tries to rewrite the response, the client still gets its error event, the body is untouched. What changes is that the failure is there to be found when something goes looking.

A canceled request is the consumer hanging up, not a failure, and still returns nil.

What this does not fix

The status stays 200 even with the error returned, because it genuinely already went out on the wire. Log-based alerting can catch this; status-code metrics cannot, and would need a counter emitted from the handler itself. Worth knowing when the dashboards get built — "2xx rate is fine" will not tell you the firehose is failing.

Tests

Both go through request-logging middleware rather than asserting on the handler's return value, since being visible to that machinery is the property the change exists for:

test with the change with return nil
TestFirehoseRouteReportsAStoreFailure PASS FAIL — "a store failure was invisible to the request logger"
TestFirehoseRouteDoesNotReportACanceledRequest PASS PASS (unchanged behaviour)

The failure test also asserts the status is 200, documenting that the status cannot carry this, and that event: error still reaches the client.

Clean: go build ./..., GOWORK=off go vet ./..., gofmt -l ., go mod tidy (no diff), GOWORK=off go test ./....


🤖 Generated with Claude Code

https://claude.ai/code/session_01AW65SG2vbQZ8X4wyzLkmVK


Generated by Claude Code

firehoseHandler wrote an error event and returned nil, so a stream that
died on a store failure looked to echo exactly like one that reached the
end of the data.

That matters for observability that does not exist yet. Echo's request
logger derives what it reports from two things: the error the handler
returns, and the committed response status. This handler defeated both.
It returned nil, and it commits 200 before the first record, so the
status is fixed by the time anything can fail. Standing up the usual
middleware later would have logged a database outage as a clean 200 with
no error, and RED metrics would have counted it as a success -- the
signal was erased in the handler, below the layer that would look for it.

Returning the error costs nothing today: with no middleware registered,
echo routes it to DefaultHTTPErrorHandler, which returns immediately on
a committed response. The client still gets its error event and the body
is untouched. What changes is that the failure is now there to be found
when something goes looking.

A canceled request is the consumer hanging up, not a failure, and still
returns nil.

The status stays 200 even so, because it genuinely already went out on
the wire. Log-based alerting can catch this; status-code metrics cannot,
and would need a counter emitted from the handler itself.

Tested through request-logging middleware rather than on the handler's
return value, since being visible to that machinery is the property the
change exists for. The failure test fails against the previous nil.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AW65SG2vbQZ8X4wyzLkmVK
@Peeja
Peeja merged commit 1778ec0 into main Sep 18, 2026
7 checks passed
fil-forge-bot Bot added a commit to fil-forge/infra-central that referenced this pull request Sep 18, 2026
Published from fil-forge/swarf#19

- Digest: `sha256:6554c9c71caff72fc099c0578edcf9084b8bee7a2747f6161160b9faba9062f5`
- Commit: fil-forge/swarf@1778ec0
- Publish run: https://github.com/fil-forge/swarf/actions/runs/35355909923

Merging applies [`terraform/envs/dev/apps`](https://github.com/fil-forge/infra-central/tree/main/terraform/envs/dev/apps) with no further confirmation.

Co-authored-by: fil-forge-bot[bot] <318653112+fil-forge-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants