Fix agent sampling rates being ignored when sampling rules are configured

[mhlidd]

What Does This Do

Reconnects the Datadog Agent's adaptive sampling rates to the tracer when sampling rules are configured.

Introduces Sampler.agentSampler() — a default interface method returning the RateByServiceTraceSampler instance that should receive agent-published rates (null if the sampler doesn't use them):

• RateByServiceTraceSampler.agentSampler() returns this — it's its own source of agent rates.
• RuleBasedTraceSampler.agentSampler() delegates to its fallback sampler's agentSampler() (if the fallback is a Sampler), so agent rates keep reaching the inner RateByServiceTraceSampler it wraps.
• WriterFactory registers whichever sampler sampler.agentSampler() returns with DDAgentApi, instead of checking sampler instanceof RemoteResponseListener. RuleBasedTraceSampler no longer needs to implement RemoteResponseListener and forward onResponse itself.
• CoreTracer captures sampler.agentSampler() once at construction and reuses that same instance on every sampler rebuild (Sampler.Builder.forConfig(config, traceConfig, agentSampler)), replacing the old instanceof RateByServiceTraceSampler / instanceof RuleBasedTraceSampler cascade (agentSamplerOf) with a single polymorphic call.

Motivation

When any sampling rule or a default rate is configured, Sampler.Builder.forConfig returns a RuleBasedTraceSampler that delegates to an inner RateByServiceTraceSampler for spans matching no rule. That inner sampler was never subscribed to agent rates, so its rate table stayed at its constructor default of 1.0 for the life of the JVM: every rule miss was kept, stamped _dd.agent_psr=1.0 with the AGENT_RATE mechanism, regardless of what the agent computed.

Remote configuration compounds this. A change to the trace sample rate or sampling rules rebuilds the sampler in CoreTracer.ConfigSnapshot, but addResponseListener runs once at writer construction against initialSampler and is never re-run, so rebuilt samplers were disconnected too. Three broken cases:

Scenario	Before
Rules present at startup	Never subscribed; dead from JVM start
No rules at startup, rules arrive via remote config	Subscribed sampler is no longer the one consulted
Remote config removes all rules	New instance, also unsubscribed

In practice this is masked whenever a default rate is set, because build then appends an AlwaysMatchesSamplingRule and the fallback is unreachable. It becomes visible when rules are configured without a catch-all: the service loses its adaptive backstop entirely and a traffic spike is ingested at 100% until someone intervenes.

Every other tracer applies agent rates on a rule miss:

Tracer	Rule miss uses agent rate?	Survives a remote config rule change?
Go	yes	yes, prioritySampler is never rebuilt
Node	yes	yes, configure() leaves _samplers alone
Python	yes	yes, set_sampling_rules only reassigns rules
Ruby	yes	wiring survives via a delegator; rates repopulate
.NET	yes	yes, caches and replays rates into each rebuild
Java	no	no

This PR takes the Go/Node/Python shape: never rebuild the agent rate sampler, so nothing has to be re-registered or replayed and no learned rates are lost.

The behavior dates to #1102 (Nov 2019), which introduced rule based sampling. The instanceof registration predates it by ~22 months (c1f9f4fc, Jan 2018), when RateByServiceSampler was the only RemoteResponseListener and the check was correct. Adding a wrapper sampler silently stopped it matching. Remote config rebuilding arrived later still in #5466 (Jun 2023).

Additional Notes

• agentSampler() is a first-class part of the Sampler contract (default-null) rather than a set of instanceof special cases scattered across WriterFactory and CoreTracer. Any current or future composite Sampler gets the right behavior for free by overriding it, and callers never need to know the concrete sampler type.
• RuleBasedTraceSampler's fallbackSampler field is typed PrioritySampler, not Sampler, so agentSampler() still needs one instanceof Sampler check to recover the agent-rate instance through the fallback — this is a legitimate type-narrowing check, not a leftover of the old approach, and is covered by agentSamplerIsNullWhenFallbackDoesNotUseAgentRates.
• The existing 2-argument Sampler.Builder.forConfig and 5-argument RuleBasedTraceSampler.build are unchanged in behavior — they still default to a fresh RateByServiceTraceSampler when no agent sampler is supplied — so external callers and tests are unaffected.
• Not included, worth a separate ticket: RateByServiceTraceSampler stamps _dd.agent_psr=1.0 with mechanism AGENT_RATE even when no agent response has ever been received, making "the agent said keep everything" indistinguishable from "no rates ever arrived". Go, Node, Python and .NET all guard this (suppressing the tag or using the DEFAULT mechanism until the first response). This masked the present bug in telemetry. Changing it affects span tags across all configurations and needs its own discussion plus system-tests alignment.
• Not included, worth a separate ticket: WriterFactory decides what to register with DDAgentApi once, at writer construction, from whatever sampler shape exists at that moment. If the tracer starts with priority sampling disabled/forced and no rules (so sampler.agentSampler() is null at construction), nothing gets registered — and if Remote Config later introduces sampling rules or a trace sample rate, the rebuilt RuleBasedTraceSampler falls back to a brand-new, still-unregistered RateByServiceTraceSampler. This predates this PR (the same gap exists in the prior commit, just reached through the old instanceof/2-arg-forConfig path); closing it means always eagerly constructing and registering one canonical RateByServiceTraceSampler regardless of the initial sampler's shape, which is a bigger change than this fix's scope.

Testing

10 new tests:

• RuleBasedSamplerAgentRatesTest (8) — agent rates forwarded to the fallback; matched rules unaffected by agent rates; a default rate still bypasses the fallback; a supplied agent sampler is used as the fallback and returned directly when no rules exist; rates learned before a rebuild still apply after it; agentSampler() is null when the fallback isn't a Sampler; an unreported service is still kept at 1.0 until the agent responds.
• WriterFactoryTest (2) — WriterFactory registers sampler.agentSampler() with DDAgentApi when non-null, using a real JavaTestHttpServer round-trip; skips registration (and still handles responses cleanly) when agentSampler() returns null.
• TracingConfigPollerTest.samplerRebuiltByRemoteConfigStillAppliesAgentRates (1, pre-existing from the original fix, unchanged in intent) — drives a real remote config update through the poller, asserts the rebuilt sampler wraps the same agent sampler instance, and that a span matching no rule is dropped at the rate the agent published.

Both behavioral tests were confirmed to fail against the unfixed code and pass with the fix.

Full :dd-trace-core:test: 4465 tests, 1 failure — PendingTraceBufferTest.bufferFullYieldsImmediateWrite, the same order-dependent flake in this class noted in earlier runs of this suite (a different method in the same class failed then), unrelated to this change. spotlessCheck clean.

Contributor Checklist

• Format the title according to the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any other useful labels
• Avoid using close, fix, or any linking keywords when referencing an issue
  Use solves instead, and assign the PR milestone to the issue
• Update the CODEOWNERS file on source file addition, migration, or deletion
• Update public documentation with any new configuration flags or behaviors
• Once approved, use merge queue to merge the PR

Jira ticket: [PROJ-IDENT]

🤖 Generated with Claude Code

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.78 s	14.63 s	[+0.2%; +1.9%] (maybe worse)
startup:insecure-bank:tracing:Agent	13.67 s	13.62 s	[-0.6%; +1.3%] (no difference)
startup:petclinic:appsec:Agent	17.01 s	16.92 s	[-0.2%; +1.2%] (no difference)
startup:petclinic:iast:Agent	16.86 s	16.95 s	[-1.5%; +0.5%] (no difference)
startup:petclinic:profiling:Agent	16.59 s	16.78 s	[-2.2%; -0.0%] (maybe better)
startup:petclinic:sca:Agent	16.94 s	16.75 s	[+0.1%; +2.1%] (maybe worse)
startup:petclinic:tracing:Agent	16.06 s	16.16 s	[-1.6%; +0.3%] (no difference)

Commit: 6dbce547 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[mcculls]

@DataDog review

[Copilot]

🟡 Changes recommended

Remote configuration can create an unregistered agent-rate fallback when the tracer starts without a local sampler, so unmatched spans still ignore agent-published rates.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This pull request reconnects Datadog Agent adaptive sampling rates to rule-based samplers and preserves them across remote-configuration rebuilds.

Changes:

• Adds polymorphic Sampler.agentSampler() delegation.
• Registers the resolved agent sampler with DDAgentApi.
• Reuses the agent sampler during sampler rebuilds.
• Adds regression coverage for sampling, writer registration, and remote configuration.

File summaries

File	Summary
dd-trace-core/src/test/java/datadog/trace/core/TracingConfigPollerTest.java	Tests remote-config sampler rebuilding and agent-rate behavior.
dd-trace-core/src/test/java/datadog/trace/core/DDCoreJavaSpecification.java	Adds shared rate-response test data.
dd-trace-core/src/test/java/datadog/trace/common/writer/WriterFactoryTest.java	Tests agent-sampler listener registration.
dd-trace-core/src/test/java/datadog/trace/common/sampling/RuleBasedSamplerAgentRatesTest.java	Tests rule-based forwarding and fallback behavior.
dd-trace-core/src/main/java/datadog/trace/core/CoreTracer.java	Reuses the agent sampler during configuration rebuilds.
dd-trace-core/src/main/java/datadog/trace/common/writer/WriterFactory.java	Registers the delegated agent sampler.
dd-trace-core/src/main/java/datadog/trace/common/sampling/Sampler.java	Adds the agent-sampler contract and builder support.
dd-trace-core/src/main/java/datadog/trace/common/sampling/RuleBasedTraceSampler.java	Delegates agent-rate handling to its fallback sampler.
dd-trace-core/src/main/java/datadog/trace/common/sampling/RateByServiceTraceSampler.java	Identifies itself as the agent sampler.

Review details

• Files reviewed: 9/9 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[datadog-official]

More details

The sampler contract keeps one agent-rate sampler registered and reuses it when remote configuration rebuilds the rule sampler. Read-only source review identifies no concrete regression in the changed paths.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 297d2d6 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 297d2d6573

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[datadog-official]

More details

The change keeps one agent-rate sampler during rule changes. The writer registers that same sampler, so learned rates continue to apply when no rule matches.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 297d2d6 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}