Skip to content

Commit 6a7601f

Browse files
committed
Flush the DDIntakeWriter track instead of adding a second LLM Obs writer
The DDAgentWriter branch's dedicated LLM Obs DDIntakeWriter duplicated the LLM Obs track that the DD_INTAKE_WRITER_TYPE branch already builds whenever llmObsEnabled is true (Agent.java defaults to MultiWriter:DDIntakeWriter,DDAgentWriter for that case), so every span was sent twice. Verified live on Lambda: the pre-fix build sent LLMOBS/v2 payloads twice per invocation. The original bug this branch was fixing -- spans silently dropped in Lambda -- was actually caused by the DDIntakeWriter branch never setting alwaysFlush, so the periodic flush timer rarely beat the execution environment freezing after the handler returns. Verified live: without alwaysFlush, 0/8 invocations delivered a span; with it, 8/8 did, one send each.
1 parent a0ba04e commit 6a7601f

1 file changed

Lines changed: 9 additions & 20 deletions

File tree

‎dd-trace-core/src/main/java/datadog/trace/common/writer/WriterFactory.java‎

Lines changed: 9 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -134,19 +134,27 @@ public static Writer createWriter(
134134
}
135135
}
136136

137-
Writer remoteWriter;
137+
RemoteWriter remoteWriter;
138138
if (DD_INTAKE_WRITER_TYPE.equals(configuredType)) {
139139
final TrackType trackType = DDIntakeTrackTypeResolver.resolve(config);
140140
final RemoteApi remoteApi =
141141
createDDIntakeRemoteApi(config, commObjects, featuresDiscovery, trackType);
142142

143+
// In a serverless environment the execution environment can freeze as soon as the handler
144+
// returns, before this writer's periodic flush timer next fires -- flush synchronously so
145+
// buffered events (e.g. LLM Observability spans) aren't lost when that happens.
146+
boolean alwaysFlush =
147+
config.isAgentConfiguredUsingDefault()
148+
&& ServerlessInfo.get().isRunningInServerlessEnvironment();
149+
143150
DDIntakeWriter.DDIntakeWriterBuilder builder =
144151
DDIntakeWriter.builder()
145152
.addTrack(trackType, remoteApi)
146153
.prioritization(prioritization)
147154
.healthMetrics(healthMetrics)
148155
.monitoring(commObjects.monitoring)
149156
.singleSpanSampler(singleSpanSampler)
157+
.alwaysFlush(alwaysFlush)
150158
.flushIntervalMilliseconds(flushIntervalMilliseconds);
151159

152160
if (config.isCiVisibilityEnabled()) {
@@ -215,25 +223,6 @@ public static Writer createWriter(
215223
}
216224

217225
remoteWriter = builder.build();
218-
219-
// DDAgentWriter only speaks the regular trace protocol, so LLM Observability spans
220-
// (tagged with DDSpanTypes.LLMOBS) need their own track sent via the EVP proxy -- without
221-
// this, they're silently dropped by the agent/extension instead of reaching LLM Obs
222-
// intake. Flush this track synchronously too when the primary writer is (i.e. in a
223-
// serverless environment), so spans aren't lost when the execution environment freezes.
224-
if (config.isLlmObsEnabled()) {
225-
final RemoteApi llmObsApi =
226-
createDDIntakeRemoteApi(config, commObjects, featuresDiscovery, TrackType.LLMOBS);
227-
final DDIntakeWriter llmObsWriter =
228-
DDIntakeWriter.builder()
229-
.addTrack(TrackType.LLMOBS, llmObsApi)
230-
.healthMetrics(healthMetrics)
231-
.monitoring(commObjects.monitoring)
232-
.alwaysFlush(alwaysFlush)
233-
.flushIntervalMilliseconds(flushIntervalMilliseconds)
234-
.build();
235-
remoteWriter = new MultiWriter(new Writer[] {remoteWriter, llmObsWriter});
236-
}
237226
}
238227

239228
return remoteWriter;

0 commit comments

Comments
 (0)