Skip to content

feat: add ClickHouse read pool checkout failure telemetry - #3996

Merged
Ziinc merged 1 commit into
mainfrom
adammokan/o11y-2427-handle-db_connection-connection_error-telemetry-for
Sep 17, 2026
Merged

Ziinc merged 1 commit into
mainfrom
adammokan/o11y-2427-handle-db_connection-connection_error-telemetry-for

Conversation

@amokan

@amokan amokan commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Overview

DBConnection sheds read pool checkouts under saturation (:queue_timeout) or when a queued request's deadline expires. Nothing counts them today, and the shed wait lands in read_pool.checkout.pool_time as if a connection had been obtained.

DBConnection already hands every failed checkout to the :log callback we pass on Ch.query/4, with connection_time nil. This matches that shape and emits a counter from the callback, where backend_id and the read cluster label are already in scope.

Key Changes

  • New handle_read_pool_log/3 head matches a ConnectionError result with connection_time: nil and emits [:logflare, :clickhouse, :read_pool, :checkout_error] tagged backend_id, read_cluster, reason. A successful checkout always records a :prepare event first, so post-checkout errors never match
  • The head skips the latency measurements and slow-checkout warning, so shed requests no longer inflate pool_time
  • logflare.clickhouse.read_pool.checkout_error declared as a sum. reason mirrors DBConnection.ConnectionError.reason
  • checkout_error and query_error descriptions state the overlap: a queue_timeout appears in both

Risks / Considerations

  • Do not sum the two series checkout_error is per attempt and the pool-health signal; query_error is per query and the user-facing failure rate
  • pool_time p99 drops under saturation Shed checkouts were the largest samples. Re-baseline any alerts on it

@amokan
amokan requested a review from djwhitt September 15, 2026 20:28
@amokan
amokan requested a review from Ziinc September 15, 2026 20:28
@Ziinc
Ziinc merged commit 039000b into main Sep 17, 2026
26 of 27 checks passed
@Ziinc
Ziinc deleted the adammokan/o11y-2427-handle-db_connection-connection_error-telemetry-for branch September 17, 2026 12:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants