Skip to content

fix(vertexai): record the accumulated text for async streaming responses - #4657

Closed
Eapp1e wants to merge 1 commit into
traceloop:mainfrom
Eapp1e:fix/vertexai-async-stream-completion
Closed

Eapp1e wants to merge 1 commit into
traceloop:mainfrom
Eapp1e:fix/vertexai-async-stream-completion

Conversation

@Eapp1e

@Eapp1e Eapp1e commented Oct 25, 2026 •

Copy link
Copy Markdown

What

_abuild_from_streaming_response accumulates the text of every streamed chunk into complete_response, but it passes the async generator object itself to handle_streaming_response:

async for item in response:
    complete_response += str(item.text)
    ...
handle_streaming_response(span, event_logger, llm_model, response, token_usage)

set_response_attributes then writes that object into gen_ai.completion.0.content. The span ends up holding a memory address (<async_generator object ... at 0x...>) instead of the model output, and the SDK logs Invalid type <class 'async_generator'> for attribute value on every call. With events enabled (use_legacy_attributes=False) the async path emits no gen_ai.choice event at all, because emit_response_events only handles str and GenerationResponse.

The sync sibling _build_from_streaming_response already does it right — it passes complete_response (lines 159-161).

How

Pass the accumulated text, matching the sync path:

handle_streaming_response(
    span, event_logger, llm_model, complete_response, token_usage
)

Tests

New tests/test_streaming_attributes.py drives _abuild_from_streaming_response with an in-memory AsyncGenerator of chunks and asserts the exported span:

  • gen_ai.completion.0.content == "Hello world!" (before the fix this is the generator repr),
  • gen_ai.completion.0.role == "assistant".

Config.use_legacy_attributes is pinned with monkeypatch so the assertion is mode-independent. The whole package suite passes locally: 18 passed.

The expectation matches tests/disabled_test_bison.py:134, which asserts gen_ai.completion.0.content == "".join(response) for the same async streaming path.

Prepared with the help of an AI assistant.

Summary by CodeRabbit

  • Bug Fixes
    • Streaming responses now record the accumulated completion text rather than an async-generator representation.
    • Completion attributes now include the assistant role, improving the accuracy of streaming response details in telemetry.

_abuild_from_streaming_response collected every chunk's text into
complete_response and then passed the async generator object itself to
handle_streaming_response. set_response_attributes wrote that object into
gen_ai.completion.0.content, so the span carried an object repr instead of
the model output and the SDK logged an invalid attribute type on every
call. With events enabled no gen_ai.choice record was emitted at all,
because emit_response_events only handles str and GenerationResponse.

Pass complete_response, as the synchronous path already does, and cover the
async streaming span with a unit test.
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@coderabbitai

coderabbitai Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: b8ac6195-bcc8-4f39-8f2f-c2b96a578cfb
Commits

Reviewing files that changed from the base of the PR and between 8d9c93f and e44efe9.

Files selected for processing: 2
  • packages/opentelemetry-instrumentation-vertexai/opentelemetry/instrumentation/vertexai/__init__.py
  • packages/opentelemetry-instrumentation-vertexai/tests/test_streaming_attributes.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

Walkthrough

The async Vertex AI streaming path now passes accumulated response text to handle_streaming_response. New tests check that streamed completion content is concatenated and that the recorded completion role is "assistant".

Changes: Vertex AI async streaming attributes

Layer / File(s) Summary
Pass accumulated text and verify attributes
packages/opentelemetry-instrumentation-vertexai/opentelemetry/instrumentation/vertexai/__init__.py, packages/opentelemetry-instrumentation-vertexai/tests/test_streaming_attributes.py
The async streaming path passes accumulated response text to handle_streaming_response. Tests check concatenated completion content and the "assistant" role.

Priority: Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: luizdmm

Merge Risk: Minimal · up to e44ef

The async streaming fix and its focused attribute tests are mergeable with no identified actionable risk.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main change: recording accumulated text for asynchronous Vertex AI streaming responses.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@Eapp1e

Eapp1e commented Oct 25, 2026

Copy link
Copy Markdown
Author

This project requires signing the CLA, and I cannot complete that on behalf of my account, so I am closing this PR rather than leaving it blocked. The analysis and patch stay available in the branch if someone able to sign wants to take them over.

@Eapp1e Eapp1e closed this Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants