Repository navigation
fix(report-api): cap static file transfer time at 1 minute - #161
Merged
Merged
Conversation
A slow client kept a /v1/static proxy transfer open until Cloud Run's 3600s request timeout, which surfaced as a 502. The existing 120s timeout only fires on an idle socket, so a client that keeps reading slowly never hits it. Abort any transfer after STATIC_MAX_TRANSFER_MS (default 5 min).
Signed-off-by: Max Ostapenko <1611259+max-ostapenko@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Limits how long a single
/v1/static/*transfer can run: 1 minute by default, configurable withSTATIC_MAX_TRANSFER_MS.Problem
On 09-27 a slow bot (
AtlasBot) downloaded/v1/static/almanac/ebooks/web_almanac_2022_en.pdfthrough the GCS proxy. The request stayed open until Cloud Run's 3600s timeout killed it, and it was logged as a502. The proxy's existingreq.setTimeout(60000)only fires when the socket is idle, so a client that keeps reading slowly never hits it.Change
proxyReportsFilestarts a timer when streaming begins. If the transfer is still running when it fires, the timer logs aWARNINGwith the object path and destroys the GCS read stream.pipelinethen closes the response. Headers have already been sent at that point, so the client sees a truncated download rather than an error status.unref'd so it never keeps the process alive.The largest files served today are the Web Almanac ebooks (~10 MB). Those take seconds on a normal connection, so 1 minute only affects clients that are very slow or stalled.
Tests
report-api: 121/121 tests pass, lint clean.