Skip to content

chore(tests): run AI Guard and LLMObs with uv - #20098

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 4 commits into
mainfrom
codex/migrate-ai-llm-suitespec
Sep 8, 2026
Merged

chore(tests): run AI Guard and LLMObs with uv#20098
gh-worker-dd-mergequeue-cf854d[bot] merged 4 commits into
mainfrom
codex/migrate-ai-llm-suitespec

Conversation

@mabdinur

@mabdinur mabdinur commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Description

  • Declare all AI Guard and LLMObs test environments in suitespec.
  • Run these suites through uv while preserving Riot commands, dependencies, Python coverage, environment variables, and lock identities.
  • Keep the current CI Visibility test-skipping environment on the affected LLMObs integrations.

The primary regression gate is test_suitespec_matches_riot. It compares every migrated environment with Riot, including commands, dependencies, environment variables, and lock hashes.

Testing

  • Passed test_suitespec_matches_riot locally
  • Smoke-tested AI Guard API and LLMObs Pydantic v1 environments
  • Generated GitLab jobs for all enabled AI Guard and LLMObs suites
  • Passed scripts/lint suitespec-check and scripts/lint checks

Risks

Low. This changes test environment orchestration only; Riot definitions and lock files remain as the parity oracle.

Additional Notes

@mabdinur mabdinur added the changelog/no-changelog A changelog entry is not required for this PR. label Sep 7, 2026
@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Sep 7, 2026 •
edited
Loading

Copy link
Copy Markdown

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

No remaining files require a CODEOWNERS review.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Sep 7, 2026 •
edited
Loading

Copy link
Copy Markdown

Circular import analysis

⚠️ Existing circular imports

There are 2 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector
ddtrace.appsec._asm_request_context -> ddtrace.appsec._iast._iast_request_context_base -> ddtrace.appsec._iast._iast_env -> ddtrace.appsec._iast.reporter -> ddtrace.appsec._exploit_prevention.stack_traces -> ddtrace.appsec._asm_request_context

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Sep 7, 2026 •
edited
Loading

Copy link
Copy Markdown

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 230 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 230 highest severity) 
ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=134)
ddtrace.llmobs._telemetry -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)
ddtrace.debugging._signal.model -×-> ddtrace.trace  (product:debugging -> product:tracing, score=132)
ddtrace.profiling.collector.pytorch -×-> ddtrace.trace  (product:profiling -> product:tracing, score=132)
ddtrace.internal.ci_visibility.recorder -×-> ddtrace.trace  (product:ci_visibility -> product:tracing, score=132)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

@mabdinur mabdinur mentioned this pull request Sep 7, 2026
Merged
@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Sep 7, 2026 •
edited by datadog-official Bot
Loading

Copy link
Copy Markdown
Contributor

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ea13539 | Docs | View more details | Give us feedback!

Base automatically changed from codex/declare-all-suitespec to main September 7, 2026 13:02
@mabdinur
mabdinur force-pushed the codex/migrate-ai-llm-suitespec branch from f214b14 to 84ff668 Compare September 7, 2026 13:52
@mabdinur
mabdinur marked this pull request as ready for review September 7, 2026 14:47
@mabdinur
mabdinur requested review from a team as code owners September 7, 2026 14:47
@mabdinur
mabdinur requested a review from avara1986 September 7, 2026 14:47
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 7, 2026 •
edited
Loading

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-07T14:56:43.209343Z 84ff668 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed Sep 7, 2026
View reviewed changes

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 84ff66858f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tests/llmobs/suitespec.yml
gh-worker-dd-mergequeue-cf854d Bot pushed a commit that referenced this pull request Sep 7, 2026
@mabdinur
chore(tests): run security suites with uv (#20063)
f87bd63 
## Description

- Declare AppSec, IAST, and SCA test environments in suitespec.
- Run those security suites through uv.
- Preserve Riot commands, dependencies, Python coverage, environment variables, and lock identities.
- Keep the existing xdist configuration for AppSec threat suites.

The primary regression gate is [test_suitespec_matches_riot](https://github.com/DataDog/dd-trace-py/blob/main/tests/contrib/integration_registry/test_riotfile.py). It compares every migrated environment with Riot, including commands, dependencies, environment variables, and lock hashes.

## Testing

- Passed test_suitespec_matches_riot locally
- Generated the GitLab test configuration successfully
- Smoke-tested AppSec, IAST, and SCA environments with uv
- Passed scripts/lint suitespec-check and scripts/lint checks

## Risks

Low. This changes test environment orchestration only. Riot definitions and lock files remain as the parity oracle.

## Additional Notes

- Stacked on #20012
- AI Guard and LLMObs are migrated separately in #20098
- No release note is needed for this test-only change


Co-authored-by: munir.abdinur <munir.abdinur@datadoghq.com>
@mabdinur
mabdinur requested a review from Yun-Kim September 8, 2026 15:09
mabdinur
mabdinur commented Sep 8, 2026
View reviewed changes
Comment thread tests/llmobs/suitespec.yml
@mabdinur
mabdinur force-pushed the codex/migrate-ai-llm-suitespec branch 2 times, most recently from deb42b5 to a8647c5 Compare September 8, 2026 16:09
@mabdinur
mabdinur force-pushed the codex/migrate-ai-llm-suitespec branch from a8647c5 to ea13539 Compare September 8, 2026 17:02
@mabdinur

mabdinur commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

/merge -f --reason transient pip download connection reset before pytest started; PR CI and affected suites are green

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Sep 8, 2026 •
edited
Loading

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-09-08 19:21:25 UTC ℹ️ Start processing command /merge -f --reason transient pip download connection reset before pytest started; PR CI and affected suites are green

2026-09-08 19:21:30 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 0s (p90).

2026-09-08 19:21:39 UTC ℹ️ MergeQueue: This merge request was merged

Warning

This change was merged without running any pre merge CI checks

Reason: transient pip download connection reset before pytest started; PR CI and affected suites are green

@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit a672a68 into main Sep 8, 2026
354 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the codex/migrate-ai-llm-suitespec branch September 8, 2026 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments
@avara1986 avara1986 avara1986 approved these changes
@Yun-Kim Yun-Kim Yun-Kim approved these changes

Assignees

No one assigned

Labels

changelog/no-changelog A changelog entry is not required for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@mabdinur @avara1986 @Yun-Kim