Skip to content

Allow outlier detection to distinguish client deadline expiry from other cancellations #13001

Closed

Description

@kishorekasi

When a client call deadline expires before the server responds, ClientCall.Listener receives DEADLINE_EXCEEDED, but the subchannel's ClientStreamTracer receives CANCELLED.

The path appears to be:

ClientCallImpl.CancellationHandler calls RetriableStream.cancel(DEADLINE_EXCEEDED).
RetriableStream commits a NoopClientStream as the logical winner.
The physical substream is cancelled with CANCELLED_BECAUSE_COMMITTED.
The subchannel tracer receives that physical stream status as CANCELLED.
Relevant code:

RetriableStream.cancel()
CANCELLED_BECAUSE_COMMITTED
CommitTask cancelling physical streams
If the server instead returns DEADLINE_EXCEEDED, the tracer receives DEADLINE_EXCEEDED directly.

Desired outlier-detection behavior
We would like to count both of these as endpoint failures:

The server returns DEADLINE_EXCEEDED.
The client deadline expires while waiting for that endpoint.
We would not like to count:

Explicit application cancellation.
Context cancellation unrelated to deadline expiry.
Cancellation of non-winning retry or hedging attempts.
At present, these locally initiated cases appear indistinguishable to an attempt-level ClientStreamTracer because they can all arrive as CANCELLED.

Question
Is there a supported way for a load-balancer policy or ClientStreamTracer to distinguish client deadline expiry from other locally initiated cancellations?

If not, would grpc-java consider exposing the cancellation origin, or otherwise preserving enough information for an outlier-detection policy to make this distinction?

I noticed that grpc/proposal#558 currently proposes excluding locally initiated deadline expiry from outlier-detection counting. Is that intended to be a fixed semantic, or could policies support counting deadline expiry when it is considered a useful endpoint-health signal?

Activity

  1. AgraVator commented on Aug 24, 2026

    @AgraVator
    Contributor

    Thanks for bringing this up. We recently had a discussion about this exact issue regarding Outlier Detection (gRFC A50) and how to handle hedging and local cancellations.

    To answer your questions:

    Distinguishing cancellations: There is currently no clean way for a ClientStreamTracer to distinguish between a client-side deadline expiry and other local cancellations (like hedging or application cancellation), as the retry layer translates them into CANCELLED.

    Exposing the origin: We do not plan to expose the cancellation origin to allow counting client deadline expiries as failures.

    gRFC A50 / Proposal #558 intent: The proposal to exclude locally-initiated deadline expiries is indeed intended to be a fixed cross-language semantic.

    Why this direction?
    The maintainers agreed that relying on status codes for LB accounting is brittle. Furthermore, because it is difficult to definitively prove whether a client-side deadline expiry is due to a bad server or a network issue, the consensus is to treat all locally initiated cancellations (from "above") equally.

    Moving forward, the plan is to introduce a clean cancellation hook (e.g., ClientStreamTracer.cancelled()) so the load-balancing layer can simply ignore any call canceled locally before the server responds. Therefore, client-side deadline expiries will not be counted as endpoint failures.

  2. kannanjgithub commented on Aug 28, 2026

    @kannanjgithub
    Contributor

    Closing this issue as we have communicated our stance about deadline cancellations. Feel free to comment to reopen if there is more to discuss on this point.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions