feat(api): add optional observability adapter

[harvhan]

Pull Request Summary

Add an optional module that can replace API logging, tracing, and metrics. A selected, installed adapter owns instrumentation. When the selector is unset or the module is absent, the API uses its built-in telemetry.

Changes

• Name the selection predicate uses_observability_adapter() so its purpose is explicit.
• Use two callbacks: initialize(app) after app assembly and shutdown() after application cleanup. Importing the adapter does not configure logging.
• Skip native providers, custom metrics, StatsD, and PostgreSQL/Redis collectors in adapter mode. Keep the existing native startup path otherwise.
• Let images require installed OpenTelemetry SDK setup before native auto-instrumentation, so bundled adapter packages alone do not activate it. Native custom metrics and injected SDK setup remain available.
• Preserve request IDs through responses, cached HTTP clients, streams, and errors. Use AsyncExitStack for cleanup after startup or shutdown failures.
• Load the SQLAlchemy engine factory after adapter initialization so its tracing hooks apply.
• Reuse the native console handler on repeated make_logger() calls and module reloads, preserving other handlers and the existing log format.

Test Plan

• Full local suite: 1,352 unit and integration tests passed with real PostgreSQL, MongoDB, and Redis. Includes the new bootstrap guard, native metric preservation, adapter/lifecycle selection, request IDs, and logging regressions.
• Integration: two real Uvicorn workers passed HTTP, request-ID, per-process initialization, and shutdown checks.
• Ruff, formatting, and whitespace checks passed.

Follow-up

Application log-call cleanup remains in #435, after this PR. Existing log message text is unchanged here. No logging helpers, lint baseline, or logging CI job is added. The adapter interface intentionally does not preserve native custom metrics when an adapter is selected. Review dashboard and alert consumers of auth_cache.access, auth_cache.eviction, and agent_run_schedule.temporal_op before activation. Request-ID validation does not prevent spoofing; deployments requiring service-controlled IDs must overwrite caller-supplied IDs at trusted ingress.

Confidence Score: 5/5

The PR appears safe to merge. No new blocking issue was found.

What we checked:

• Dependency update breaks supported setups: Python 3.12 Linux and Apple Silicon macOS have matching wheels. Intel macOS has a documented Rust or container path, and the existing Fernet calls still match the tested API.

Summary

The API can now hand logging, tracing, and metrics to an optional installed observability module. It also keeps request IDs connected through middleware, errors, streams, and outgoing HTTP calls while preserving the built-in telemetry path when no adapter is selected.

• Adds adapter initialization and shutdown callbacks around the completed FastAPI app.
• Disables native providers and collectors when an adapter owns observability.
• Adds request-ID forwarding and safer cleanup during startup and shutdown.
• Reuses native console handlers and documents the adapter contract.

Diagram

sequenceDiagram
    participant Worker as API worker
    participant Selector as observability selector
    participant Adapter as optional adapter
    participant Native as built-in telemetry
    participant Deps as dependencies
    participant Client as HTTP client

    Worker->>Selector: load configured module
    alt adapter is installed
        Worker->>Adapter: initialize(app)
        Worker->>Deps: start without native collectors
    else adapter is absent
        Worker->>Native: start auto-instrumentation and metrics
        Worker->>Deps: start with native collectors
    end
    Client->>Worker: request with optional x-request-id
    Worker->>Worker: keep or create request ID
    Worker-->>Client: response with x-request-id
    Worker->>Deps: close clients and dependencies
    alt adapter mode
        Worker->>Adapter: shutdown()
    else native mode
        Worker->>Native: flush metrics
    end

Loading

_{Reviews (9) · Last reviewed commit: "Merge branch 'main' into harvhan/runtime..."}

[stephen-wang24]

This turns off every custom instrument in adapter mode, not just the ones sgp-obs replaces.

db_metrics is fine — sgp-obs emits the same db.client.* family. But cache_metrics (hit / miss_expired / miss_absent) and schedule_metrics (Temporal create/update/delete) have no sgp-obs equivalent, so those series simply stop when the adapter is selected.

The PR description calls this intentional, which I'm not arguing with — but it doesn't say what currently reads them. Worth enumerating the dashboards and alert rules that consume cache_* and schedule_* before anyone sets AGENTEX_OBSERVABILITY_MODULE on a real environment. If nothing does, say so in the adapter doc so the next person doesn't have to re-derive it; if something does, those need re-creating on sgp-obs' meter rather than dropping.

[stephen-wang24]

This changes the trust boundary. Previously the ID was always minted here; now an inbound header wins. A caller can send the same x-request-id on every request and flatten log correlation for the whole service.

That's the right call behind the mesh — Envoy stamps the header, and sgp-obs' own request_id module documents exactly this assumption ("the shape check prevents log injection, not spoofing"). But observability_adapter.md describes the validator and not the assumption behind it, so there's no way to tell from the docs whether this is safe on a directly-exposed deployment.

One sentence saying inbound IDs are trusted because the mesh owns the header would cover it.

[stephen-wang24]

Worth a comment on why this import is function-local: SQLAlchemyInstrumentor patches create_async_engine, so a module-level import binds the unpatched symbol and the adapter silently gets no SQL spans.

As written it reads like an accident, and that reasoning currently lives only in the PR description — the next cleanup pass will hoist it back to the top of the file and the failure will be invisible.