solana/rpc: record endpoint rate-limit headroom per method

[bgm-malbeclabs]

Summary

A 429 from a ledger RPC endpoint is currently opaque. solana-go keeps only a code on its error types:

type HTTPError struct { Code int; err error }
type RPCError  struct { Code int; Message string; Data any }

…and discards the http.Response inside CallForInto. So the X-Ratelimit-* headers that say what the cap actually was are gone before any caller sees the error. A service can log "Too many requests for a specific RPC call" for hours without ever learning the number it is exceeding.

This adds a passive RoundTripper on the outermost transport of every retrying client this package builds, recording:

• doublezero_solana_rpc_ratelimit_method_limit{method}
• doublezero_solana_rpc_ratelimit_method_remaining{method}

Labelled by JSON-RPC method because that is how the limits are enforced — per method, per source IP, over a rolling window (confirmed with Triton). An unattributed number would not be actionable, and a global request-rate graph does not show this at all: one method can be shedding while total throughput looks healthy.

Motivating incident

A lake indexer activity was rate-limited on getTransaction for 4.5 hours (malbeclabs/lake#753, #757). Throughout, the cap was unknowable from our side:

• The headers were never captured — discarded at the solana-go layer, as above. Confirmed by searching 7 days of logs across every lake-* and dz-* namespace: zero lines contain X-Ratelimit-Method-Limit, -Remaining, or -Tier.

• The vendor's suggested workaround ("grab them off one of the 429 responses") was not available either. A live getTransaction against the mainnet ledger endpoint returns no X-Ratelimit-* headers at all on a 200:

  HTTP/2 200
  content-type: application/json; charset=utf-8
  x-rpc-node: lb-pit5
  access-control-allow-origin: *
  alt-svc: h3=":443"; ma=86400
  

So these gauges may only populate on the 429 path with the current endpoint configuration. That is still exactly when the number matters, and it costs nothing in the meantime — see below. If the headers later appear on successful responses, this starts showing continuous headroom with no further change.

Cost when unused

• Header check first. resp.Header.Get on two keys; if both are absent it returns immediately. No body inspection, no allocation, no series.
• Body read only when there is something to record, and through req.GetBody() so the body the transport actually sends is untouched. Bounded to 512 bytes — JSON-RPC puts method near the front.
• Never alters a request or response, and never fails a call. A body it cannot parse, or a request without GetBody, is silently skipped.
• Batch requests are labelled batch, mirroring the batchLabel the retry layer already uses, since a batch carries a mix of methods.

Testing Verification

tools/solana/pkg/rpc/ratelimit_test.go:

• RecordsHeadersByMethod — headers land in the right gauges under the right method label, the response is unaltered, and the inner transport still receives the complete, unmodified body (the property that makes reading via GetBody correct).
• NoHeadersNoSeries — the cheap-exit path: an endpoint reporting nothing produces no series and no body inspection. This is the path the DZ mainnet ledger takes today.
• BatchIsLabelledBatch — a batch is not misattributed to whichever method happens to be first.
• MalformedBodyIsSkipped — observation is best-effort; an unparseable body does not fail the request.

Full go test ./tools/solana/... passes with no regressions; gofmt and go vet clean.

Note on scope

This deliberately does not add an Options.Transport hook. That was the other way to solve it, but it would push the work onto every caller and let them replace the tuned transport (gzip, connection pools, dial and request timeouts) by accident. Rate-limit classification already lives in this package — isRetryableHTTPStatus handles 429 — so surfacing why belongs here too, once, for everyone.