Repository navigation
fix: report a failed firehose stream to the caller of the handler - #19
Merged
Merged
Conversation
firehoseHandler wrote an error event and returned nil, so a stream that died on a store failure looked to echo exactly like one that reached the end of the data. That matters for observability that does not exist yet. Echo's request logger derives what it reports from two things: the error the handler returns, and the committed response status. This handler defeated both. It returned nil, and it commits 200 before the first record, so the status is fixed by the time anything can fail. Standing up the usual middleware later would have logged a database outage as a clean 200 with no error, and RED metrics would have counted it as a success -- the signal was erased in the handler, below the layer that would look for it. Returning the error costs nothing today: with no middleware registered, echo routes it to DefaultHTTPErrorHandler, which returns immediately on a committed response. The client still gets its error event and the body is untouched. What changes is that the failure is now there to be found when something goes looking. A canceled request is the consumer hanging up, not a failure, and still returns nil. The status stays 200 even so, because it genuinely already went out on the wire. Log-based alerting can catch this; status-code metrics cannot, and would need a counter emitted from the handler itself. Tested through request-logging middleware rather than on the handler's return value, since being visible to that machinery is the property the change exists for. The failure test fails against the previous nil. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AW65SG2vbQZ8X4wyzLkmVK
fil-forge-bot Bot
added a commit
to fil-forge/infra-central
that referenced
this pull request
Sep 18, 2026
Published from fil-forge/swarf#19 - Digest: `sha256:6554c9c71caff72fc099c0578edcf9084b8bee7a2747f6161160b9faba9062f5` - Commit: fil-forge/swarf@1778ec0 - Publish run: https://github.com/fil-forge/swarf/actions/runs/35355909923 Merging applies [`terraform/envs/dev/apps`](https://github.com/fil-forge/infra-central/tree/main/terraform/envs/dev/apps) with no further confirmation. Co-authored-by: fil-forge-bot[bot] <318653112+fil-forge-bot[bot]@users.noreply.github.com>
This was referenced Sep 19, 2026
Draft
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[From Claude:]
One line in
firehoseHandler, plus tests. Independent of #18 — that one is client-side, this is the server handler.The problem
firehoseHandlerwrites an error event and thenreturn nil(app.go:321), so a stream that died on a store failure looks to echo exactly like one that reached the end of the data.There is no logging or metrics machinery in the repo today to notice, so nothing is broken right now. The reason to change it now is that adding that machinery later would not pick these up. Echo's request logger derives what it reports from exactly two things (
request_logger.go:353onward, v4.15.0):This handler defeats both. It returns
nil, andresponse.WriteHeader(http.StatusOK)(app.go:313) commits the status before the first record, so the status is fixed before anything can fail.I measured what standard middleware sees today, with
RequestLoggerWithConfig{LogStatus, LogError, HandleError}in front of the real handler and a store that fails mid-stream:The database is down, the consumer is told, and the request logs as a clean 200. Same two inputs feed RED metrics, so it would count as a success. The signal is erased in the handler, below the layer that would go looking for it — so this is not something the observability work can fix on its own later.
The change
Return the error instead of
nil. Same setup, after:It costs nothing today. With no middleware registered, echo routes the error to
DefaultHTTPErrorHandler, which returns immediately on a committed response (echo.go:428) — nothing tries to rewrite the response, the client still gets its error event, the body is untouched. What changes is that the failure is there to be found when something goes looking.A canceled request is the consumer hanging up, not a failure, and still returns
nil.What this does not fix
The status stays 200 even with the error returned, because it genuinely already went out on the wire. Log-based alerting can catch this; status-code metrics cannot, and would need a counter emitted from the handler itself. Worth knowing when the dashboards get built — "2xx rate is fine" will not tell you the firehose is failing.
Tests
Both go through request-logging middleware rather than asserting on the handler's return value, since being visible to that machinery is the property the change exists for:
return nilTestFirehoseRouteReportsAStoreFailureTestFirehoseRouteDoesNotReportACanceledRequestThe failure test also asserts the status is 200, documenting that the status cannot carry this, and that
event: errorstill reaches the client.Clean:
go build ./...,GOWORK=off go vet ./...,gofmt -l .,go mod tidy(no diff),GOWORK=off go test ./....🤖 Generated with Claude Code
https://claude.ai/code/session_01AW65SG2vbQZ8X4wyzLkmVK
Generated by Claude Code