feat(http): add max connection age for HTTP/2 server connections - #4618
feat(http): add max connection age for HTTP/2 server connections#4618ergdevops wants to merge 4 commits into
Conversation
Pooled proxy-to-proxy HTTP/2 connections have no idle timeout or age bound: peers keep them alive indefinitely (keepalives prevent closure; MAX_IDLE_CONNS_PER_ENDPOINT only caps the count), and each held connection retains its receive-buffer high-water mark - up to the connection window - for its entire lifetime. In a mesh where a large caller fleet held ~300 connections into each server pod, we measured ~700KB retained per connection after ~300 rps load bursts (~200MB of proxy RSS per pod), released only when peer pods restarted. This adds LINKERD2_PROXY_INBOUND_SERVER_HTTP2_MAX_CONNECTION_AGE (duration; unset preserves current behavior). When set, the inbound HTTP/2 serve loop arms a timer as an additional select branch alongside the existing drain/teardown triggers and reuses the same graceful shutdown (GOAWAY) path: in-flight streams complete and clients reconnect transparently. A deterministic per-connection jitter of up to +10% (derived from the client's ephemeral port, avoiding a new dependency) prevents synchronized shutdown storms. Shutdowns are logged at INFO with the client address. Signed-off-by: ergdevops <115262164+ergdevops@users.noreply.github.com>
At scale this fires once per connection per age period, which is too chatty for INFO; operators who want per-shutdown visibility can enable it via the proxy log filter. Signed-off-by: ergdevops <115262164+ergdevops@users.noreply.github.com>
- Drop the doc comment on ServerParams::max_connection_age to match the struct's style. - Rename pa/peer to peer_addr for clarity and consistency. - Deduplicate the H2 serve loop's shutdown handling with a Teardown enum classifying the completion branches (ClientClosed / Drain / Graceful), as suggested in review. Signed-off-by: ergdevops <115262164+ergdevops@users.noreply.github.com>
12a80c6 to
a758233
Compare
Like every other SERVER_HTTP2 setting, max connection age parses under both the inbound and outbound server prefixes via the shared parse_server; make that symmetry explicit with a test exercising the real variable names. Signed-off-by: ergdevops <115262164+ergdevops@users.noreply.github.com>
ergdevops
changed the title
|
@thesw4rm With the symmetry addressed and tests covering both prefixes, I believe the feature is complete and the PR title now matches the actual scope of the change — but happy to take any further feedback. Separately, since you've contributed here before: could you help me understand the cadence the linkerd maintainers typically follow for reviewing and merging feature PRs like this? I have this PR and one other (#4613) waiting on maintainer review, and I'd like to set my expectations right — and do whatever I can from my side to make them easy to review. |
|
@ergdevops I have never contributed here before haha. I'm just going around reviewing PRs |
@adleong apologies for the tag but could you please help me with this request, thank you. |
2 participants
Addresses linkerd/linkerd2#15623.
Problem
Pooled proxy-to-proxy HTTP/2 connections currently live forever: TCP and HTTP/2 keepalives keep them healthy indefinitely,
MAX_IDLE_CONNS_PER_ENDPOINT(default 10,000) only caps the count, the HTTP/1 pool idle timeout doesn't apply to them, and the outbound discovery idle timeout only fires when a service receives no traffic at all. As far as we can tell there is no setting that closes a pooled connection for being idle or bounds its lifetime (checkedlinkerd/app/src/env.rsonmainand back throughv2.311.0).This matters for memory: each held server-side connection retains its receive-buffer high-water mark — bounded by the connection window (1MB default) — for its entire lifetime. In our mesh (arm64 EKS), a large caller fleet holds ~300 pooled connections into each server pod; after ~300 rps load bursts we measured ~700KB retained per connection, i.e. ~200MB of proxy RSS per pod that was only ever released when peer pods restarted. Verified via
tcp_open_total/tcp_close_totalaccounting on the admin endpoint — the pooling itself works as designed; the cost is the unbounded buffer lifetime.Change
Adds a
max_connection_agesetting to the shared HTTP/2ServerParams, configurable per server prefix like every otherSERVER_HTTP2setting:LINKERD2_PROXY_INBOUND_SERVER_HTTP2_MAX_CONNECTION_AGE(duration) — the inbound proxy-to-proxy listener, where pooled peer connections accumulate;LINKERD2_PROXY_OUTBOUND_SERVER_HTTP2_MAX_CONNECTION_AGE(duration) — the outbound (app-facing) listener, for symmetry with the other paired settings.Unset (the default) preserves current behavior exactly. The admin server builds its params with
Default::default()and is unaffected regardless of environment.When set, the HTTP/2 serve loop arms a timer as an additional
select!branch alongside the existing drain/teardown triggers, reusing the samegraceful_shutdown()(GOAWAY) path: no new streams are accepted, in-flight streams run to completion, then the connection closes and the client reconnects transparently — the identical sequence every pod drain already exercises. A deterministic per-connection jitter of up to +10% (derived from the client's ephemeral port; no new dependency) prevents synchronized shutdown storms. Each shutdown is logged at DEBUG with the client address.Includes an integration test covering: requests served normally before the age elapses, an in-flight stream completing across the age boundary, and the connection closing once drained — plus an env test asserting the setting parses independently under both server prefixes.
Results from our environment (15m age, production-like mesh)
Similar in spirit to gRPC's
MAX_CONNECTION_AGE; follows the same params/env plumbing pattern as #4542.