chore(agents): add dd-apm-sdk-review skill with two starter rules and cases by robertomonteromiguel · Pull Request #6301 · DataDog/dd-trace-rb

robertomonteromiguel · 2026-09-10T15:00:43Z

What does this PR do?

Adds the shared dd-apm-sdk-review pre-push skill to this repo, plus two small Ruby-owned rules and two matching eval cases, so the team can see the whole loop and start writing the next ones.

Concretely:

.agents/skills/dd-apm-sdk-review/ — verbatim copy of dd-apm-sdk-review-core. Do not edit here.
.agents/dd-apm-sdk-review-overrides/ — this repo's layer. Two starter rules:
- conventions — DATADOG_ENV not ENV; Datadog::Core::Utils::Time.now not Time.now (from AGENTS.md).
- security — never write an API key into a span tag or a log line.
.llm-validation/ — two cases, one per rule. Copy either to add the next one. How-to is in .llm-validation/README.md.
.gitlab-ci.yml — reusable "llm validation" job from ddoghq/llm-validation-platform (same pin as js/java).

Claude discovers the skill via a symlink at .claude/skills/dd-apm-sdk-review.

How to review this

Skip .agents/skills/dd-apm-sdk-review/ — exact copy of the core repo.
Read the two overrides under .agents/dd-apm-sdk-review-overrides/reviewers/. These are the part this repo owns.
Read the two cases in .llm-validation/suites/dd-apm-sdk-review.yaml. Each case would fail if its rule disappeared.
Skim .llm-validation/README.md — that is the contribution guide we want people to follow.

Motivation:

js and java already have this skill plus a real eval suite. Ruby has no repo-owned review rules yet. This PR is intentionally tiny: two rules, two cases, the CI job. The goal is to make the next rule a 5-minute copy-paste, not a design review.

Same gate as dd-trace-js#10137 and dd-trace-java#12409.

Change log entry

None. Internal agent-review tooling. Not customer-visible.

How to add the next rule

Extend or add a file under .agents/dd-apm-sdk-review-overrides/reviewers/.
Copy a case in .llm-validation/suites/dd-apm-sdk-review.yaml.
List the new id under presets.gate.cases in .llm-validation/config.yaml.
Open a PR.

How to test the change?

From the repo root, with Docker and ddtool:

export LLMVAL_IMAGE=registry.ddbuild.io/ci/llm-validation-platform/llmval:latest
export LLMVAL_AUTH_HEADER="$(ddtool auth token rapid-ai-platform --datacenter us1.staging.dog --http-header)"
docker run --rm -e LLMVAL_AUTH_HEADER -v "$PWD:/repo" "$LLMVAL_IMAGE" \
  --repo /repo --base-sha master --level minimum --runs 1

--level minimum is one case. --level gate is both.

Made with Cursor

… cases Give the Ruby tracer the same pre-push review skill as js/java, plus two small repo-owned rules and matching eval cases so the team can copy the pattern and grow the suite. Co-authored-by: Cursor <cursoragent@cursor.com>

datadog-official · 2026-09-10T15:00:53Z

✨ Unblock PR with BitsAI

⚠️ Warnings

❌ Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 23 Pipeline jobs failed

Build gem | Build gem (dev) — 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

Build gem | Build gem (final) — 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

Check Pull Request CI Status | all-jobs-are-green — 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

View all 23 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: c22626b | Docs | View more details | Give us feedback!}

Drop the how-to-add-a-rule sections from the starter overrides and point harness-less reviewers at the skill rule files without running the skill. Co-authored-by: Cursor <cursoragent@cursor.com>

pr-commenter · 2026-09-10T15:48:05Z

LLM Validation

LLM Validation Gate — dd-apm-sdk-review

✅ PASS

No blocking-case regressions; the quality change is within noise (baseline/candidate confidence intervals overlap).

Analysis

Changed instruction file(s): .agents/skills/dd-apm-sdk-review/SKILL.md, .agents/skills/dd-apm-sdk-review/reviewers/_common.md, .agents/skills/dd-apm-sdk-review/reviewers/coherence.md, .agents/skills/dd-apm-sdk-review/reviewers/correctness.md, .agents/skills/dd-apm-sdk-review/reviewers/security.md, .agents/skills/dd-apm-sdk-review/reviewers/design.md, .agents/skills/dd-apm-sdk-review/reviewers/performance.md, .agents/skills/dd-apm-sdk-review/reviewers/maintainability.md, .agents/skills/dd-apm-sdk-review/reviewers/conventions.md, .agents/skills/dd-apm-sdk-review/reviewers/cross-sdk.md, .agents/skills/dd-apm-sdk-review/reviewers/report-template.md, .agents/dd-apm-sdk-review-overrides/repo-context.md, .agents/dd-apm-sdk-review-overrides/reviewers/conventions.md, .agents/dd-apm-sdk-review-overrides/reviewers/security.md.

No safety or blocking-case regressions across 2 case(s). Overall pairwise win-rate 50% [50%–50%], quality +3.1 — see the verdict above for whether that clears the noise band.

Results

Pairwise win-rate: 50% [50%–50%] — candidate's share of blind comparisons (90% CI; spanning 50% = no clear difference)
Overall quality: 87.8 → 90.9 (/100, +3.1)
Bad signals introduced (advisory): 0
Candidate criteria coverage (advisory): 7/7 (100%) — expected_criteria the candidate met; does not affect the gate
Blocking-case regressions: 0

Cases

Case	Mode	Quality Δ	Win-rate (90% CI)	Safety
rb-conventions-env-and-time	block	0.0	50% [50%–50%]	ok
rb-security-secret-into-tag	block	+6.3	50% [50%–50%]	ok

_{Per-dimension scores, token usage, latency, and estimated cost are in the CI job logs.}

Keep the local harness paragraph in AGENTS.md; Codex loads the mirrored core file instead of an inline lens list.

robertomonteromiguel · 2026-09-11T06:58:53Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b87f68c317

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Three parent traversals resolved above the repo root and left the link dangling; match js/java and point at ../../.agents/skills. Co-authored-by: Cursor <cursoragent@cursor.com>

robertomonteromiguel · 2026-09-11T07:52:00Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fc92bd38aa

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

The shared P0 contract requires a source location; the snippet now lives at a synthetic path so a correct review does not have to invent one.

Mirror the core skill: performance, design, and conventions reviews use the generic files when this repo has not written an override yet.

robertomonteromiguel · 2026-09-11T09:09:39Z

@codex review

chatgpt-codex-connector · 2026-09-11T09:13:32Z

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: c22626b97d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

robertomonteromiguel · 2026-09-15T07:40:13Z

Superseded by the stacked split: core copy in #6306, Ruby overrides in #6307. Closing this combined landing so review stays on those two PRs.

robertomonteromiguel added the AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos label Sep 10, 2026

chore(agents): match js/java Codex review fallback in AGENTS.md

575fab0

Drop the how-to-add-a-rule sections from the starter overrides and point harness-less reviewers at the skill rule files without running the skill. Co-authored-by: Cursor <cursoragent@cursor.com>

chore(agents): point Codex at the shared no-harness review contract

b87f68c

Keep the local harness paragraph in AGENTS.md; Codex loads the mirrored core file instead of an inline lens list.

chatgpt-codex-connector Bot reviewed Sep 11, 2026

View reviewed changes

Comment thread .claude/skills/dd-apm-sdk-review Outdated

Comment thread .agents/skills/dd-apm-sdk-review/SKILL.md

Comment thread .agents/skills/dd-apm-sdk-review/SKILL.md

Comment thread .agents/dd-apm-sdk-review-overrides/reviewers/security.md

chore(agents): fix Claude skill symlink to the in-repo core copy

fc92bd3

Three parent traversals resolved above the repo root and left the link dangling; match js/java and point at ../../.agents/skills. Co-authored-by: Cursor <cursoragent@cursor.com>

chatgpt-codex-connector Bot reviewed Sep 11, 2026

View reviewed changes

Comment thread .agents/skills/dd-apm-sdk-review/review-without-harness.md

Comment thread .llm-validation/suites/dd-apm-sdk-review.yaml

Comment thread .agents/skills/dd-apm-sdk-review/reviewers/performance.md Outdated

robertomonteromiguel added 2 commits September 11, 2026 10:59

test(llm-validation): give the security P0 case a file:line anchor

83c55da

The shared P0 contract requires a source location; the snippet now lives at a synthetic path so a correct review does not have to invent one.

chore(agents): treat missing dd-apm-sdk-review overrides as optional

c22626b

Mirror the core skill: performance, design, and conventions reviews use the generic files when this repo has not written an override yet.

This was referenced Sep 11, 2026

chore(agents): add verbatim dd-apm-sdk-review skill copy #6306

Draft

chore(agents): add dd-trace-rb overrides for dd-apm-sdk-review #6307

Draft

robertomonteromiguel closed this Sep 15, 2026

Navigation Menu

chore(agents): add dd-apm-sdk-review skill with two starter rules and cases - #6301

chore(agents): add dd-apm-sdk-review skill with two starter rules and cases#6301
robertomonteromiguel wants to merge 6 commits into
masterfrom
robertomonteromiguel/dd-apm-sdk-review-core-overrides

robertomonteromiguel commented Sep 10, 2026

Uh oh!

datadog-official Bot commented Sep 10, 2026 •

edited

Loading

Uh oh!

pr-commenter Bot commented Sep 10, 2026

Uh oh!

robertomonteromiguel commented Sep 11, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

robertomonteromiguel commented Sep 11, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

robertomonteromiguel commented Sep 11, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 11, 2026

Uh oh!

robertomonteromiguel commented Sep 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

robertomonteromiguel commented Sep 10, 2026

Uh oh!

datadog-official Bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

pr-commenter Bot commented Sep 10, 2026

LLM Validation

LLM Validation Gate — dd-apm-sdk-review

✅ PASS

Analysis

Results

Cases

Uh oh!

robertomonteromiguel commented Sep 11, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

robertomonteromiguel commented Sep 11, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

robertomonteromiguel commented Sep 11, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 11, 2026

Uh oh!

robertomonteromiguel commented Sep 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

datadog-official Bot commented Sep 10, 2026 •

edited

Loading