feat(sampling): Support for OpenTelemetry consistent tracestate sampling by MilanGarnier · Pull Request #12397 · DataDog/dd-trace-java

MilanGarnier · 2026-09-03T13:05:58Z

What Does This Do

Adds OpenTelemetry consistent probability-sampling behavior on top of the
ot tracestate parsing introduced by #12405.

Validates and normalizes rv and th.
Generates probability state for loaded agent rates and sampling rules.
Preserves inherited randomness and unknown fields across overrides.
Keeps limiter outcomes distinct from raw probability decisions.
Adds focused parsing, propagation, continuation, and sampling tests.

Motivation

Make Java tracer sampling decisions interoperable with OpenTelemetry
consistent probability sampling and provide the state needed for downstream
tracestate propagation.

Additional Notes

depends on refactor(propagation): Parse OpenTelemetry tracestate member #12405 which introduced the OtelTracestate class.
system-tests will be enabled in test(java): enable ot.th/ot.rv tracestate sampling scenarios [java@milan.garnier/ot.th] system-tests#7649
OTLP export optimizations (which should already work with this PR, but it can be optimized) will follow in a separate change.

Contributor Checklist

Format the title according to the contribution guidelines.
Assign the required type: and comp: labels.

Jira ticket: APMAPI-2171

datadog-datadog-prod-us1-2 · 2026-09-03T13:20:46Z

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 54.75% (-4.30%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: ac097ca | Docs | View more details | Give us feedback!}

dd-octo-sts · 2026-09-03T13:27:22Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.90 s	14.68 s	[+0.5%; +2.5%] (maybe worse)
startup:insecure-bank:tracing:Agent	13.60 s	13.74 s	[-2.0%; -0.1%] (maybe better)
startup:petclinic:appsec:Agent	17.49 s	17.41 s	[-0.6%; +1.5%] (no difference)
startup:petclinic:iast:Agent	17.39 s	17.50 s	[-1.3%; +0.2%] (no difference)
startup:petclinic:profiling:Agent	17.23 s	17.25 s	[-1.4%; +1.0%] (no difference)
startup:petclinic:sca:Agent	16.84 s	17.30 s	[-6.9%; +1.6%] (no difference)
startup:petclinic:tracing:Agent	16.58 s	16.65 s	[-1.3%; +0.4%] (no difference)

Commit: ac097ca2 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

MilanGarnier · 2026-09-08T14:48:29Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a0176c99c0

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 629892d492

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

datadog-datadog-prod-us1-2

Compound extraction can emit a W3C sampled flag that conflicts with the propagated OTel probability threshold when propagation styles have different sampling decisions.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 3c81345 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

MilanGarnier · 2026-09-14T16:24:32Z

Since the PR is a bit big (~1100 lines of feature, 1000 of tests), I've rewritten the history into a set of commits that should each be independently reviewable and testable.

MilanGarnier · 2026-09-14T16:25:09Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e9d42f767a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Reuse the immutable empty sampling state across propagation tags. Cache effective W3C tracestate by immutable sampling state identity. Size W3C encoding buffers from the captured sampling state.

datadog-datadog-prod-us1-2

A matching traceparent and Datadog priority can keep an OTel threshold that gives the opposite sampling decision. Reinjection then sends conflicting sampling data.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit d324e16 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

datadog-datadog-prod-us1-2 · 2026-09-15T09:57:24Z

+      SamplingState currentState = samplingState;
+      this.samplingState =
+          newSamplingState(
+              currentState.getSamplingPriority(),
+              tracestate,
+              otelTraceState,


Remove a threshold that conflicts with the sampled flag

A downstream OTel sampler can drop a trace that traceparent marks as sampled.

Assertion details

Input: Use a sampled traceparent with tracestate: dd=s:1,ot=rv:00000000000000;th:8.

Expected: Remove the threshold when the parsed OTel state conflicts with the final sampling priority.

Actual: Extraction keeps th:8. Injection then sends a sampled traceparent with an ot state that means drop.

Suggested change

SamplingState currentState = samplingState;

this.samplingState =

newSamplingState(

currentState.getSamplingPriority(),

tracestate,

otelTraceState,

SamplingState currentState = samplingState;

int samplingPriority = currentState.getSamplingPriority();

if (otelTraceState != null

&& samplingPriority != PrioritySampling.UNSET

&& !otelTraceState.isConsistentWith(samplingPriority > 0)) {

otelTraceState = otelTraceState.withoutThreshold();

}

this.samplingState =

newSamplingState(

samplingPriority,

tracestate,

otelTraceState,

Was this helpful? React 👍 or 👎
_{🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest} _{· Open Bits AI session}

…value

datadog-datadog-prod-us1-2

The change can keep an inherited OpenTelemetry threshold that conflicts with a new local sampling decision. It also sends threshold values with extra trailing zeroes.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit a51831b · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

datadog-datadog-prod-us1-2 · 2026-09-15T12:36:49Z

      }
-      this.samplingPriority = samplingPriority;
-      if (samplingPriority > 0) {
+      OtelTraceState nextOtelTraceState = getOtelTraceState();


Keep local decisions consistent with inherited thresholds

A downstream tracer can make a different sampling decision from the decision in the trace flags.

Assertion details

Input: An inbound ot state contains rv and th. The tracer then makes a local probability decision while the sampling priority permits the update.

Expected: The code must use the inherited random value with a threshold that matches the new local decision. It can remove the threshold if it cannot make the values consistent.

Actual: The code keeps the inherited threshold for most local probability decisions. The inherited threshold can show the opposite decision.

Was this helpful? React 👍 or 👎
_{🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest} _{· Open Bits AI session}

datadog-datadog-prod-us1-2 · 2026-09-15T12:36:49Z

+        }
+      } else if (startsWith(raw, start, end, THRESHOLD_KEY)) {
+        int candidateStart = start + THRESHOLD_KEY.length();
+        if (thresholdStart < 0 && isLowerHex(raw, candidateStart, end, 1, 14)) {


Remove trailing zeroes from inherited thresholds

A strict downstream tracer can reject the ot state and stop the consistent sampling decision.

Assertion details

Input: An inbound ot state contains a valid threshold with extra trailing hexadecimal zeroes, such as th:8000.

Expected: The parser must send the standard form th:8 and keep one zero when the threshold is zero.

Actual: The parser accepts a threshold such as th:8000 and sends the same value again.

Was this helpful? React 👍 or 👎
_{🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest} _{· Open Bits AI session}

duplicate dd and probably more things + regression test

MilanGarnier added type: feature Enhancements and improvements comp: context propagation Trace context propagation tag: ai generated Largely based on code generated by an AI or LLM labels Sep 3, 2026

MilanGarnier changed the title ~~Support OpenTelemetry tracestate sampling~~ feat(sampling): Support for OpenTelemetry consistent tracestate sampling Sep 3, 2026

MilanGarnier force-pushed the milan.garnier/ot.th branch from 3643548 to 2f90ba6 Compare September 3, 2026 13:58

MilanGarnier changed the base branch from master to milan.garnier/preserve-deterministic-sampler-rate September 3, 2026 13:59

Base automatically changed from milan.garnier/preserve-deterministic-sampler-rate to master September 3, 2026 18:45

MilanGarnier force-pushed the milan.garnier/ot.th branch from a168627 to 9e2422a Compare September 4, 2026 09:29

MilanGarnier mentioned this pull request Sep 4, 2026

refactor(propagation): Parse OpenTelemetry tracestate member #12405

Merged

3 tasks

MilanGarnier changed the base branch from master to milan.garnier/otel-tracestate-refactor September 4, 2026 09:30

MilanGarnier mentioned this pull request Sep 4, 2026

test(java): enable ot.th/ot.rv tracestate sampling scenarios [java@milan.garnier/ot.th] DataDog/system-tests#7649

Open

MilanGarnier force-pushed the milan.garnier/ot.th branch 5 times, most recently from 7a1ddea to a0176c9 Compare September 8, 2026 14:26

chatgpt-codex-connector Bot reviewed Sep 8, 2026

View reviewed changes

MilanGarnier marked this pull request as ready for review September 10, 2026 09:02

MilanGarnier requested review from a team as code owners September 10, 2026 09:02

MilanGarnier requested review from mhlidd and removed request for a team September 10, 2026 09:02

MilanGarnier force-pushed the milan.garnier/otel-tracestate-refactor branch from db321c0 to 1c02400 Compare September 10, 2026 09:02

MilanGarnier force-pushed the milan.garnier/ot.th branch from 3c81345 to 629892d Compare September 10, 2026 09:02

chatgpt-codex-connector Bot reviewed Sep 10, 2026

View reviewed changes

Comment thread dd-trace-core/src/main/java/datadog/trace/core/propagation/HttpCodec.java Outdated

Comment thread dd-trace-core/src/main/java/datadog/trace/core/propagation/ptags/PTagsFactory.java Outdated

Comment thread dd-trace-core/src/main/java/datadog/trace/core/DDSpan.java Outdated

datadog-datadog-prod-us1-2 Bot reviewed Sep 10, 2026

View reviewed changes

Comment thread dd-trace-core/src/main/java/datadog/trace/core/propagation/ptags/PTagsFactory.java Outdated

MilanGarnier force-pushed the milan.garnier/ot.th branch from d5a6721 to eb4c156 Compare September 11, 2026 13:28

MilanGarnier force-pushed the milan.garnier/ot.th branch from 7772c8c to cd47a1a Compare September 14, 2026 09:14

MilanGarnier removed this pull request from stack #12406 September 14, 2026 13:01

Base automatically changed from milan.garnier/otel-tracestate-refactor to master September 14, 2026 14:41

MilanGarnier added 3 commits September 14, 2026 17:04

feat(propagation): model otel probability sampling state

d2104b0

refactor(propagation): introduce immutable sampling snapshots

7025ae9

feat(sampling): publish probability decisions atomically

1c8a762

MilanGarnier force-pushed the milan.garnier/ot.th branch from cd47a1a to e9d42f7 Compare September 14, 2026 16:16

MilanGarnier added this pull request to stack #12487 September 14, 2026 16:17

chatgpt-codex-connector Bot reviewed Sep 14, 2026

View reviewed changes

Comment thread dd-trace-core/src/main/java/datadog/trace/core/propagation/ptags/PTagsFactory.java Outdated

Comment thread dd-trace-core/src/main/java/datadog/trace/core/propagation/ptags/W3CPTagsCodec.java Outdated

MilanGarnier force-pushed the milan.garnier/ot.th branch from fdb0a3a to 34ef4a2 Compare September 15, 2026 08:38

MilanGarnier added 6 commits September 15, 2026 11:42

feat(propagation): inject consistent otel sampling state

e470bfc

feat(otlp): export consistent otel sampling state

27cc740

perf(propagation): reduce sampling state allocations

a5a43ee

Reuse the immutable empty sampling state across propagation tags. Cache effective W3C tracestate by immutable sampling state identity. Size W3C encoding buffers from the captured sampling state.

fix(propagation): preserve locked decision maker

b5b84b2

refactor(propagation): simplify otel sampling state

de7920c

fix(propagation): reset otel state for Org Guard resampling

d324e16

MilanGarnier force-pushed the milan.garnier/ot.th branch from 34ef4a2 to d324e16 Compare September 15, 2026 09:46

MilanGarnier marked this pull request as ready for review September 15, 2026 09:47

MilanGarnier removed this pull request from stack #12487 September 15, 2026 09:47

datadog-datadog-prod-us1-2 Bot reviewed Sep 15, 2026

View reviewed changes

mcculls self-requested a review September 15, 2026 10:28

update test values across tests which were hardcoding the tracestate …

a51831b

…value

MilanGarnier requested a review from a team as a code owner September 15, 2026 12:23

MilanGarnier requested review from vandonr and removed request for a team September 15, 2026 12:23

datadog-datadog-prod-us1-2 Bot reviewed Sep 15, 2026

View reviewed changes

MilanGarnier added 2 commits September 15, 2026 15:53

remove fast path for OTel rebuildTracestate as it requires checking no

255fcb2

duplicate dd and probably more things + regression test

fix(ptags): use private lock for sampling updates

ac097ca

Uh oh!

Conversation

MilanGarnier commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

MilanGarnier commented Sep 8, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MilanGarnier commented Sep 14, 2026

Uh oh!

MilanGarnier commented Sep 14, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-prod-us1-2 Bot Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-prod-us1-2 Bot Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-prod-us1-2 Bot Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MilanGarnier commented Sep 3, 2026 •

edited

Loading

datadog-datadog-prod-us1-2 Bot commented Sep 3, 2026 •

edited

Loading

dd-octo-sts Bot commented Sep 3, 2026 •

edited

Loading