Repository navigation
feat: record time to first token on the OpenTelemetry chat span - #447
Merged
Merged
Conversation
Streamed responses now record the time to first token (in seconds) as the `gen_ai.response.time_to_first_chunk` attribute on the `chat` span. It is measured from request issuance (`chat_perform()` blocks until headers arrive) and recorded once, at the first chunk carrying non-empty text or thinking content. Ports tidyverse/ellmer#1142.
cpsievert
marked this pull request as ready for review
October 7, 2026 15:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports tidyverse/ellmer#1142.
Summary
Streamed responses now record the time to first token (in seconds) as the
gen_ai.response.time_to_first_chunkattribute on the OpenTelemetrychat <model>span.chatlas/_otel.py:record_chat_ttft_attr()sets the attribute on recording spans, given a start timestamp.chatlas/_chat.py: both sync and async streaming loops capturestream_startjust beforechat_perform()(which blocks until headers arrive, so TTFT is measured from request issuance) and record the attribute once, at the first chunk carrying non-empty text or thinking content (via the existingdisplay_text()helper).Tests: deterministic sync/async cases asserting the exact interval at the first non-empty text or thinking token, including provider-call latency and ensuring later tokens do not overwrite it, plus TTFT assertions in the VCR streaming-span test and the async stream-iteration test.
The
evalextra temporarily keeps the Anthropic SDK below 2.0 to avoid an incompatibility in Inspect's Anthropic scorer.Two small adaptations from the R implementation:
time.monotonic()is used instead of wall-clock time since this measures an elapsed interval, and the mocked empty first delta usesContentText.model_construct(text="")(as real providers do) becauseContentText.__init__normalizes empty text to"[empty string]".Verification
pytest tests/test_otel.py— all 27 tests pass, including the newtest_ttft_recorded_at_first_non_empty_tokenand the new TTFT assertions on sync/async streamed chat spans.Full local suite with CI's terminal and AWS credential setup: 1,302 passed, 44 skipped. Project-wide Pyright and Ruff checks pass.