feat(worker): add optional observability adapter

[harvhan]

The API foundation in #434 is merged; this PR targets main.

Let the platform Temporal worker use an optional telemetry adapter while preserving native startup when the module is unset or unavailable. Initialize only at the worker entry point, attach tracing once per client, and let custom Python metrics use the adapter's provider. Keep the native Core exporter and DB pool identities.

SIGTERM stops the worker before dependency cleanup and telemetry flushing. Workflow logs are replay-aware in adapter mode, and credential redaction survives logging-handler replacement. Non-JSON health responses log status and size in both modes. API schedule imports do not initialize or shut down worker telemetry.

Local validation:

• 173 focused public checks passed using the merged frozen lockfile without the private package, including API lifecycle and schedule imports.
• One real multi-process Uvicorn integration test passed.
• Paired runtime integration checks passed for real Temporal retries/replay, native fallback, and independent Core export.
• Ruff, formatting, and whitespace checks passed.

Earlier local image validation on September 18, 2026 covered the actual worker command in enabled/unset/missing modes, real Schedule dispatch, JSON logs, and idle SIGTERM exit under one second. This was a cached image with source mounts. One pre-PR review ran; its API import finding is fixed and tested. Remote CI is separate from these local results.

Activation: set the worker's AGENTEX_OBSERVABILITY_MODULE to an installed worker adapter and retain the existing Core metrics route. Remove the selector, restore prior instrumentation settings, and restart to roll back. Before activation, configure explicit pod grace and proxy egress through activity drain, dependency cleanup, and telemetry flushing; verify busy-worker scale-in with the deployed proxy. See the adapter contract. No deployment performed.

Confidence Score: 5/5

The PR appears safe to merge. No new blocking issue remains after the latest changes.

Summary

The Temporal worker can now use an optional observability adapter without changing native startup when the adapter is unset or unavailable. It adds adapter-owned tracing, Python metrics, replay-aware logs, safer health-check messages, and ordered shutdown while keeping Temporal Core metrics and database pool identities separate.

• Worker startup and shutdown now include the adapter lifecycle.
• Temporal clients and custom Python metrics can use adapter providers.
• Workflow and health-check logs avoid untrusted response details in adapter mode.
• The guide documents worker adapter setup and callbacks.

Diagram

sequenceDiagram
    participant CLI as Worker CLI
    participant Obs as Observability adapter
    participant Deps as Worker dependencies
    participant Temporal as Temporal client
    participant Core as Temporal Core metrics
    participant Worker as Temporal worker

    CLI->>Obs: initialize_worker()
    alt adapter is selected and installed
        Obs->>Obs: initialize adapter and log filters
    else adapter is unset or missing
        Obs-->>CLI: keep native telemetry
    end
    CLI->>Deps: start shared dependencies
    Deps->>Temporal: create client
    opt adapter supplies interceptors
        Obs-->>Temporal: client interceptors
    end
    opt Core metrics URL is set
        Temporal->>Core: create separate OTLP exporter
    end
    CLI->>Worker: run with 10-second shutdown grace
    Worker->>Temporal: poll agentex-server queue
    CLI-->>Worker: SIGTERM cancels main task
    Worker->>Worker: stop polling and drain activities
    CLI->>Deps: close HTTP clients and dependencies
    alt adapter mode
        CLI->>Obs: flush adapter off event loop
    else native mode
        CLI->>Core: shut down Python metrics
    end

Loading

_{Reviews (3) · Last reviewed commit: "docs(worker): clarify shutdown activatio..."}

[stephen-wang24]

This flush is registered first on the AsyncExitStack, so it unwinds last — after the 10s graceful_shutdown_timeout and after dependency close. Worth confirming the deployment actually sets an explicit terminationGracePeriodSeconds: our worker chart doesn't, so the ceiling is Kubernetes' 30s default, and with autoscaling enabled scale-in SIGTERMs busy workers, not just rollouts.

Suggestion: Pin the grace period and the service-mesh drain window so that activity drain <= mesh drain <= pod grace before enabling the adapter — a flush scheduled last is the first thing a draining sidecar or SIGKILL takes out.

[harvhan]

The deployment chart needs explicit pod-grace support and proxy drain settings that cover activity drain, dependency cleanup, and telemetry flushing. We'll document this activation prerequisite; the idle SIGTERM test does not establish safe shutdown during busy-worker scale-in.