Emit gen_ai.usage.* token metrics with LLM Observability disabled

[mz1119]

Stacked on #12450.

What Does This Do

The openai-java decorators already read the token counts off each response, but tagged them behind the llmObsEnabled guard, so an APM span carried gen_ai.operation.name, gen_ai.request.model, gen_ai.provider.name and gen_ai.application.name but no usage when LLM Observability was off.

Each count now goes through TokenUsage.set, which keeps writing the _ml_obs_metric.* tag when LLM Observability is enabled (byte-identical to today) and writes the matching gen_ai.usage.* APM metric when it is not. Covers all four endpoints: chat completions (streaming included, usage only, as dd-trace-js does), responses, embeddings and legacy completions.

Motivation

Parity with dd-trace-js, which emits gen_ai.usage.* from setGenAiApmUsageMetrics on the LLM-Observability-disabled path.

Additional Notes

_ml_obs_metric.* tags stay behind the guard, so APM spans of users who never enabled LLM Observability do not pick up LLM Observability internal tags.

The shared mock OpenAI server in the forked tests now returns real chat-completion and embedding JSON with a usage block, so the token counts have something to come from.

🤖 Generated with Claude Code

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.78 s	14.66 s	[-0.1%; +1.8%] (no difference)
startup:insecure-bank:tracing:Agent	13.64 s	13.64 s	[-0.9%; +0.8%] (no difference)
startup:petclinic:appsec:Agent	17.09 s	17.00 s	[-0.4%; +1.4%] (no difference)
startup:petclinic:iast:Agent	16.96 s	17.03 s	[-1.2%; +0.3%] (no difference)
startup:petclinic:profiling:Agent	16.16 s	16.91 s	[-8.7%; -0.1%] (maybe better)
startup:petclinic:sca:Agent	17.03 s	16.82 s	[+0.2%; +2.3%] (maybe worse)
startup:petclinic:tracing:Agent	16.17 s	16.05 s	[-0.3%; +1.7%] (no difference)

Commit: 3e59d9a2 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[mz1119]

@codex review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-15T18:53:55.332960Z	3e59d9a	Manual request
🔒 Security Review	✅ Completed 2026-09-15T18:54:40.268245Z	3e59d9a	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 3e59d9a249

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 3e59d9a249

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.