You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Allow outlier detection to distinguish client deadline expiry from other cancellations #13001
When a client call deadline expires before the server responds, ClientCall.Listener receives DEADLINE_EXCEEDED, but the subchannel's ClientStreamTracer receives CANCELLED.
The path appears to be:
ClientCallImpl.CancellationHandler calls RetriableStream.cancel(DEADLINE_EXCEEDED).
RetriableStream commits a NoopClientStream as the logical winner.
The physical substream is cancelled with CANCELLED_BECAUSE_COMMITTED.
The subchannel tracer receives that physical stream status as CANCELLED.
Relevant code:
Desired outlier-detection behavior
We would like to count both of these as endpoint failures:
The server returns DEADLINE_EXCEEDED.
The client deadline expires while waiting for that endpoint.
We would not like to count:
Explicit application cancellation.
Context cancellation unrelated to deadline expiry.
Cancellation of non-winning retry or hedging attempts.
At present, these locally initiated cases appear indistinguishable to an attempt-level ClientStreamTracer because they can all arrive as CANCELLED.
Question
Is there a supported way for a load-balancer policy or ClientStreamTracer to distinguish client deadline expiry from other locally initiated cancellations?
If not, would grpc-java consider exposing the cancellation origin, or otherwise preserving enough information for an outlier-detection policy to make this distinction?
I noticed that grpc/proposal#558 currently proposes excluding locally initiated deadline expiry from outlier-detection counting. Is that intended to be a fixed semantic, or could policies support counting deadline expiry when it is considered a useful endpoint-health signal?
Thanks for bringing this up. We recently had a discussion about this exact issue regarding Outlier Detection (gRFC A50) and how to handle hedging and local cancellations.
To answer your questions:
Distinguishing cancellations: There is currently no clean way for a ClientStreamTracer to distinguish between a client-side deadline expiry and other local cancellations (like hedging or application cancellation), as the retry layer translates them into CANCELLED.
Exposing the origin: We do not plan to expose the cancellation origin to allow counting client deadline expiries as failures.
gRFC A50 / Proposal #558 intent: The proposal to exclude locally-initiated deadline expiries is indeed intended to be a fixed cross-language semantic.
Why this direction?
The maintainers agreed that relying on status codes for LB accounting is brittle. Furthermore, because it is difficult to definitively prove whether a client-side deadline expiry is due to a bad server or a network issue, the consensus is to treat all locally initiated cancellations (from "above") equally.
Moving forward, the plan is to introduce a clean cancellation hook (e.g., ClientStreamTracer.cancelled()) so the load-balancing layer can simply ignore any call canceled locally before the server responds. Therefore, client-side deadline expiries will not be counted as endpoint failures.
Closing this issue as we have communicated our stance about deadline cancellations. Feel free to comment to reopen if there is more to discuss on this point.
Activity
AgraVator commentedon Aug 24, 2026
Thanks for bringing this up. We recently had a discussion about this exact issue regarding Outlier Detection (gRFC A50) and how to handle hedging and local cancellations.
To answer your questions:
Distinguishing cancellations: There is currently no clean way for a ClientStreamTracer to distinguish between a client-side deadline expiry and other local cancellations (like hedging or application cancellation), as the retry layer translates them into CANCELLED.
Exposing the origin: We do not plan to expose the cancellation origin to allow counting client deadline expiries as failures.
gRFC A50 / Proposal #558 intent: The proposal to exclude locally-initiated deadline expiries is indeed intended to be a fixed cross-language semantic.
Why this direction?
The maintainers agreed that relying on status codes for LB accounting is brittle. Furthermore, because it is difficult to definitively prove whether a client-side deadline expiry is due to a bad server or a network issue, the consensus is to treat all locally initiated cancellations (from "above") equally.
Moving forward, the plan is to introduce a clean cancellation hook (e.g., ClientStreamTracer.cancelled()) so the load-balancing layer can simply ignore any call canceled locally before the server responds. Therefore, client-side deadline expiries will not be counted as endpoint failures.
kannanjgithub commentedon Aug 28, 2026
Closing this issue as we have communicated our stance about deadline cancellations. Feel free to comment to reopen if there is more to discuss on this point.