chore(agents): add dd-trace-rb overrides for dd-apm-sdk-review

.agents/dd-apm-sdk-review-overrides/repo-context.md

@@ -0,0 +1,12 @@
+# Repo context — dd-trace-rb
+
+Read only by the orchestrator (Step 0 of `SKILL.md`), not by individual reviewers. Repo-specific; not part of the shared core. This whole `.agents/dd-apm-sdk-review-overrides/` folder is owned by this repo — edit it freely, unlike `.agents/skills/dd-apm-sdk-review/`, which is a verbatim copy of the shared core.
+
+## Related skills in this repo
+
+The other skills in this repo author specific things; this one is the general multi-perspective review. Cite them as authoritative for their own area, do not invoke them, and note they must not invoke this skill either:
+
+- `write-comment` — when a comment earns its place. Defer for comment-only questions.
+- `write-rbs` — RBS / Steep signatures. Defer for type-signature work.
+
+Cursor personas under `.cursor/rules/` (`code-style.mdc`, `testing.mdc`) are local editor guidance, not skills. Cite them when a conventions finding is really "the persona already said this."

.agents/dd-apm-sdk-review-overrides/reviewers/conventions.md

@@ -0,0 +1,32 @@
+Override for `reviewers/conventions.md` (in the core skill folder) — read that file first, then this.
+
+# Codebase conventions — dd-trace-rb specifics
+
+These addenda apply to `lib/**/*.rb` only. NEVER flag `ENV` or `Time.now` in
+specs, `tasks/`, gemfiles, CI, or other non-library paths.
+
+This file starts with one confirmed pattern and should grow — add the next
+one you learn from review. Do not treat it as exhaustive.
+
+The source of truth is [`AGENTS.md`](../../../AGENTS.md). Open that file for the topic under review; do not restate it here.
+
+`DATADOG_ENV` vs `ENV` is already enforced by `CustomCops/EnvUsageCop` on
+`lib/**/*` (see `.rubocop.yml`). Do not re-flag a direct `ENV` read that
+RuboCop already covers. If RuboCop did not run, report `NOT VERIFIED` for
+that check rather than inventing an ENV finding.
+
+## `Datadog::Core::Utils::Time.now`, never `Time.now`
+
+The time provider is configurable (`Core::Utils::Time.now_provider=`).
+`Time.now` in `lib/**/*.rb` is a conventions finding — tests cannot override
+it. **P1**. NEVER flag `Time.now` outside `lib/`.
+
+## Mechanical checks — run these, don't eyeball them
+
+Check-mode only. Anything that would rewrite files is the author's to run.
+
+```bash
+bundle exec rake rubocop typecheck
+```
+
+If Bundler, RuboCop, or Steep is missing, report `NOT VERIFIED (<reason>)` rather than eyeballing format.

.github/CODEOWNERS

@@ -1,7 +1,9 @@
 * @DataDog/ruby-guild
 
-# Agent review skill
+# Agent review skill and its eval suite
 /.agents/ @DataDog/ruby-guild
+/.agents/dd-apm-sdk-review-overrides/ @DataDog/ruby-guild
+/.llm-validation/ @DataDog/ruby-guild
 
 # Public Documentation
 docs/Compatibility.md @DataDog/documentation

.gitignore

@@ -71,3 +71,8 @@ lib/*.so
 
 # Superpowers brainstorming/plan output (local, not tracked)
 /docs/superpowers/
+
+# LLM Validation local/CI artifacts
+.llm-validation/results.json
+.llm-validation/report.md
+.llm-validation/details.json

.gitlab-ci.yml

@@ -17,6 +17,15 @@ include:
   - project: "DataDog/apm-reliability/apm-sdks-benchmarks"
     file: ".gitlab/ci-ruby-acme-parallel.yml"
     ref: "6e0c06b02f6e4642caec88abdb958ec32d69c6b8" # apm-sdks-benchmarks#215 merge commit
+  - project: "ddoghq/llm-validation-platform"
+    ref: &llmval_platform_sha "c331696647a78672101002ba48ccebbd41b0fae8"
+    file: "/ci/llm-validation.gitlab-ci.yml"
+
+"llm validation":
+  variables:
+    LLMVAL_PLATFORM_PROJECT: "ddoghq/llm-validation-platform"
+    LLMVAL_PLATFORM_REF: *llmval_platform_sha
+    LLMVAL_BASE_REF: "master"
 
 variables:
   RUBY_CUSTOM_IMAGE_BASE: $DOCKER_REGISTRY/ci/dd-trace-rb/custom_ruby

.llm-validation/README.md

@@ -0,0 +1,66 @@
+# LLM Validation — `dd-apm-sdk-review`
+
+This folder is how we test the review skill. It is **not** an RSpec file.
+The cases live here; the runner lives in [`ddoghq/llm-validation-platform`](https://github.com/ddoghq/llm-validation-platform).
+
+It answers: *did an edit to a review rule make the agent better or worse?*
+
+## Add a rule (this is the whole contribution)
+
+Overrides are owned by this repo. The shared core is not — never edit `.agents/skills/dd-apm-sdk-review/`.
+
+1. Create or extend a file under [`.agents/dd-apm-sdk-review-overrides/reviewers/`](../.agents/dd-apm-sdk-review-overrides/reviewers/).
+   Copy the shape of `conventions.md`: one pattern, why it matters, the fix.
+2. Add a case in [`suites/dd-apm-sdk-review.yaml`](./suites/dd-apm-sdk-review.yaml). Copy the starter case.
+   A good case is a 10-line snippet plus 2–3 `expected_criteria` that would fail if the rule disappeared.
+3. List the new case id under `presets.gate.cases` in [`config.yaml`](./config.yaml) if you want CI to run it.
+4. Open a PR. That is it.
+
+The starter case in this folder is the example. Keep new ones that short.
+
+## Layout
+
+| Path | Role |
+|---|---|
+| [`config.yaml`](./config.yaml) | Monitored instruction files, model, `--level` presets |
+| [`suites/dd-apm-sdk-review.yaml`](./suites/dd-apm-sdk-review.yaml) | Cases (one file only — the CLI errors if `suites/` has more than one YAML) |
+
+## Run locally (Docker)
+
+From the **`dd-trace-rb` repo root**:
+
+```bash
+export LLMVAL_IMAGE=registry.ddbuild.io/ci/llm-validation-platform/llmval:latest
+docker pull "$LLMVAL_IMAGE"
+
+# Offline smoke — no gateway, no Claude (1 case)
+docker run --rm -v "$PWD:/repo" "$LLMVAL_IMAGE" \
+  --repo /repo --base-sha master --level minimum --fake
+
+# Cheap real smoke — still 1 case
+export LLMVAL_AUTH_HEADER="$(ddtool auth token rapid-ai-platform --datacenter us1.staging.dog --http-header)"
+docker run --rm -e LLMVAL_AUTH_HEADER -v "$PWD:/repo" "$LLMVAL_IMAGE" \
+  --repo /repo --base-sha master --level minimum --runs 1
+
+# Gate set
+docker run --rm -e LLMVAL_AUTH_HEADER -v "$PWD:/repo" "$LLMVAL_IMAGE" \
+  --repo /repo --base-sha master --level gate --runs 1
+
+# One named case
+docker run --rm -e LLMVAL_AUTH_HEADER -v "$PWD:/repo" "$LLMVAL_IMAGE" \
+  --repo /repo --base-sha master --case rb-conventions-time-now --runs 1
+```
+
+`--level` picks **which cases** run (`minimum` = 1, `gate` = the starter case, `full` = every case).
+`--runs` only repeats those cases. Needs `ddtool` on the host for a real (non-`--fake`) run.
+
+CI includes the reusable `"llm validation"` job from the platform repo (see `.gitlab-ci.yml`).
+
+## What a pass means
+
+This is an A/B comparison, not an absolute score:
+
+- **Candidate** = the working tree. Uncommitted edits count.
+- **Baseline** = `git show <base-sha>:<file>`. A file that is not on `master` yet is treated as added.
+
+The gate fails only on a **confident regression**. Noisy changes WARN and do not block.

.llm-validation/config.yaml

@@ -0,0 +1,41 @@
+# LLM Validation Platform — dd-trace-rb config
+#
+# Starter suite for the dd-apm-sdk-review skill. One case on purpose:
+# cheap enough to re-run while you add the next rule.
+model: claude-sonnet-5
+runs: 2
+
+instruction_files:
+  - AGENTS.md
+  - .agents/skills/dd-apm-sdk-review/SKILL.md
+  - .agents/skills/dd-apm-sdk-review/reviewers/_common.md
+  - .agents/skills/dd-apm-sdk-review/reviewers/coherence.md
+  - .agents/skills/dd-apm-sdk-review/reviewers/correctness.md
+  - .agents/skills/dd-apm-sdk-review/reviewers/design.md
+  - .agents/skills/dd-apm-sdk-review/reviewers/performance.md
+  - .agents/skills/dd-apm-sdk-review/reviewers/maintainability.md
+  - .agents/skills/dd-apm-sdk-review/reviewers/conventions.md
+  - .agents/skills/dd-apm-sdk-review/reviewers/cross-sdk.md
+  - .agents/skills/dd-apm-sdk-review/reviewers/report-template.md
+  - .agents/skills/dd-apm-sdk-review/review-without-harness.md
+  - .agents/dd-apm-sdk-review-overrides/repo-context.md
+  - .agents/dd-apm-sdk-review-overrides/reviewers/conventions.md
+
+default_level: gate
+presets:
+  gate: # The starter case. Add yours here when you want CI to run it.
+    cases:
+      - rb-conventions-time-now
+    runs: 2
+  minimum: # Cheapest smoke — the conventions override.
+    cases:
+      - rb-conventions-time-now
+    runs: 2
+  full: # Every case in suites/.
+    runs: 2
+
+policy:
+  noise_threshold: 1.0
+  pairwise_win_floor: 0.45
+  blocking_fail_floor: 0.45
+  blocking_fail_ci_upper: 0.55

.llm-validation/suites/dd-apm-sdk-review.yaml

@@ -0,0 +1,34 @@
+name: dd-apm-sdk-review
+version: "0.1"
+
+# One file on purpose: the platform CLI errors if suites/ has more than one YAML.
+# One starter case. Copy the block to add the next one.
+cases:
+  - id: rb-conventions-time-now
+    files:
+      - AGENTS.md
+      - .agents/skills/dd-apm-sdk-review/reviewers/conventions.md
+      - .agents/skills/dd-apm-sdk-review/reviewers/_common.md
+      - .agents/dd-apm-sdk-review-overrides/reviewers/conventions.md
+    input: |
+      Apply this repo's conventions-review criteria (reviewers/conventions.md, its
+      .agents/dd-apm-sdk-review-overrides/reviewers/conventions.md override, _common.md, and
+      AGENTS.md) to the following code. No git checkout available — this snippet is the
+      entire change to review. Treat the snippet as
+      `lib/datadog/tracing/contrib/http/instrumentation.rb` starting at line 10 (shipped
+      code, not a spec). Direct `ENV` reads are out of scope for this lens:
+      CustomCops/EnvUsageCop already covers them.
+
+      ```ruby
+      def started_at
+        Time.now
+      end
+      ```
+    expected_criteria:
+      - "Flags `Time.now` — shipped code under lib/ must use `Datadog::Core::Utils::Time.now` so tests can override the provider."
+      - "Names `Datadog::Core::Utils::Time.now` as the replacement, not a generic 'avoid Time.now'."
+    bad_signals:
+      - "Approves the `Time.now` call."
+      - "Treats it as a style nit with no repo-specific reason."
+      - "Invents a different time-wrapper name that this gem does not use."
+      - "Flags a direct ENV read even though this snippet has none and ENV is out of scope for this lens."

AGENTS.md

@@ -160,6 +160,18 @@ docker compose run --rm tracer-2.5 bundle exec rake test:TASK_KEY
 docker compose run --rm tracer-4.0 bundle exec rake test:TASK_KEY
 ```
 
+# Review Guidelines
+
+**Local agent with a skill harness:** Run the [dd-apm-sdk-review](./.agents/skills/dd-apm-sdk-review/) skill on demand when asked. It is not required before every push. If any
+`P0` issues are reported, you must either fix them or get explicit authorization from the human you
+are working with and record the unresolved finding in the PR description (location and class of issue only — never paste secret values, tokens, credentials, or exploit details). `P1` and `P2`
+findings can be dismissed by the human.
+
+**Reviewer without a skill harness** (for example, GitHub Codex): read and follow
+`.agents/skills/dd-apm-sdk-review/review-without-harness.md`. Do not load `SKILL.md`
+or `reviewers/report-template.md`. If the diff touches `sig/` or `vendor/rbs/`,
+also follow [`.claude/skills/write-rbs/SKILL.md`](./.claude/skills/write-rbs/SKILL.md).
+
 # Pull Requests
 
 - Push branches to `DataDog/dd-trace-rb`, not forks.

spec/datadog/release_gem_spec.rb

@@ -64,6 +64,7 @@
             |\.agents
             |\.claude
             |\.cursor
+            |\.llm-validation
             |\.github
             |\.gitlab
             |\.vscode