From 3a8e20468e129662c3c8ab6e03e96407262b9baa Mon Sep 17 00:00:00 2001 From: JeremyFunk Date: Wed, 30 Sep 2026 13:58:56 +0200 Subject: [PATCH] skills(agent-tracing): trim to what an implementing agent needs - cut human-guide links, backend background, tested-version notes, restated code - drop the Go reference; other languages follow the generic steps - Do-not lists keep only silent, non-obvious mistakes not stated in the steps - inline the GenkitForMaple processor instead of pointing at the guide - OTLP header: quoted literal space everywhere (every targeted SDK accepts it) - add Cloudflare Agents and Genkit to the OpenTelemetry skill's framework list - smolagents: enable_genai_semconv is required --- skills/maple-agent-tracing-agno/SKILL.md | 29 +--- .../SKILL.md | 18 +- .../SKILL.md | 13 -- skills/maple-agent-tracing-crewai/SKILL.md | 17 +- skills/maple-agent-tracing-dspy/SKILL.md | 18 +- skills/maple-agent-tracing-genkit/SKILL.md | 125 ++++++++++++-- .../maple-agent-tracing-google-adk/SKILL.md | 18 +- .../references/typescript.md | 18 +- skills/maple-agent-tracing-haystack/SKILL.md | 27 +-- skills/maple-agent-tracing-langchain/SKILL.md | 22 +-- .../references/typescript.md | 18 +- skills/maple-agent-tracing-litellm/SKILL.md | 22 +-- .../maple-agent-tracing-llamaindex/SKILL.md | 21 +-- skills/maple-agent-tracing-mastra/SKILL.md | 21 +-- .../SKILL.md | 9 +- .../SKILL.md | 29 +--- .../maple-agent-tracing-openrouter/SKILL.md | 18 +- .../SKILL.md | 31 +--- .../references/go.md | 155 ------------------ .../SKILL.md | 18 +- .../references/python.md | 1 - .../references/typescript.md | 2 - .../maple-agent-tracing-pydantic-ai/SKILL.md | 15 +- .../maple-agent-tracing-smolagents/SKILL.md | 17 +- skills/maple-agent-tracing-spring-ai/SKILL.md | 17 +- skills/maple-agent-tracing-strands/SKILL.md | 16 +- .../SKILL.md | 14 +- skills/maple-agent-tracing/SKILL.md | 4 +- 28 files changed, 186 insertions(+), 547 deletions(-) delete mode 100644 skills/maple-agent-tracing-opentelemetry/references/go.md diff --git a/skills/maple-agent-tracing-agno/SKILL.md b/skills/maple-agent-tracing-agno/SKILL.md index d62f64321..2adab350f 100644 --- a/skills/maple-agent-tracing-agno/SKILL.md +++ b/skills/maple-agent-tracing-agno/SKILL.md @@ -7,9 +7,7 @@ description: "Trace Agno agents with Maple: installs the OpenInference Agno inst Goal: every conversation with the Agno app shows up in Maple **Agent Sessions** as exactly one session, one turn per `run()`, with transcript, model calls, tool calls (failures marked), team-member lanes, tokens and cost where the provider returns it. -Human guide with the reasoning: https://maple.dev/docs/agent-tracing/agno - -Mechanism: `openinference-instrumentation-agno` (the instrumentor Agno's own `setup_tracing()` uses) + OTel SDK + OTLP/HTTP exporter to Maple. Agno's `setup_tracing(db=...)` and `AgentOS(tracing=True)` only write to the AgentOS database; they never export OTLP. +Mechanism: `openinference-instrumentation-agno` + OTel SDK + OTLP/HTTP exporter to Maple. Agno's `setup_tracing(db=...)` and `AgentOS(tracing=True)` only write to the AgentOS database; they never export OTLP. ## Step 0: Detect versions and existing setup @@ -52,12 +50,12 @@ opentelemetry-exporter-otlp-proto-http OTEL_SERVICE_NAME= OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name= OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev # EU: https://ingest.eu.maple.dev -OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer +OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf AGNO_TELEMETRY=false ``` -The exporter appends `/v1/traces` to `OTEL_EXPORTER_OTLP_ENDPOINT`. If you use `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` instead, give the full `.../v1/traces` URL. `AGNO_TELEMETRY=false` disables Agno's anonymous usage pings (unrelated to OTel). +The exporter appends `/v1/traces` to `OTEL_EXPORTER_OTLP_ENDPOINT`. If you use `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` instead, give the full `.../v1/traces` URL. ### 2c. Tracing module @@ -81,7 +79,7 @@ AgnoInstrumentor().instrument( ) ``` -- `enable_genai_semconv=True` is REQUIRED. It dual-writes `gen_ai.*` (messages in `{role, parts}` form, usage, tool name/args/result, agent name, `gen_ai.operation.name`). Env equivalent: `OPENINFERENCE_ENABLE_GENAI_SEMCONV=true` (use it when you can't edit the `instrument()` call, e.g. AgentOS owns it). +- `enable_genai_semconv=True` is REQUIRED. Env equivalent: `OPENINFERENCE_ENABLE_GENAI_SEMCONV=true` (use it when you can't edit the `instrument()` call, e.g. AgentOS owns it). - Import `tracing` at the top of the entry point (`main.py`, `app.py`, the ASGI module), before building agents, teams or `AgentOS`. - AgentOS: `AgentOS(tracing=True)` and `setup_tracing(db=...)` skip their setup when a real `TracerProvider` is already registered, so AgentOS's traces view stops getting spans once `tracing.py` runs first. If the user wants to keep that view, add Agno's DB exporter to the same provider (use the `db` the AgentOS uses): @@ -94,8 +92,6 @@ AgnoInstrumentor().instrument( ## Step 3: Session id (one conversation = one session) -Maple groups Agno traces by `session.id` on the run span. Agno sets it from `session_id=`. - - Pass `session_id=` on EVERY `run`, `arun`, `print_response`, `aprint_response`, `continue_run`, `acontinue_run`, and on `team.run/arun` and `workflow.run/arun`. Use the id the app already stores for the chat/thread; pass `user_id=` too if available. - Without `session_id=`, Agno mints a `uuid4()` on the first run and stores it on the `Agent`/`Team` instance; every later run of that instance reuses it. A module-level shared agent then merges all users into one session. Fix it; don't rely on the default. - Do not mint a new id per request (every turn becomes its own session). @@ -180,27 +176,14 @@ Raw span check: run spans `.run` have `session.id` + `gen_ai.operati ## Known behaviours (expected; explain if the user asks) - If `setup_tracing()`/`AgentOS(tracing=True)` ran first, `set_tracer_provider` in `tracing.py` logs "Overriding of current TracerProvider is not allowed" and spans go only to the AgentOS database. A second `instrument()` call logs "Attempting to instrument while already instrumented" and its `config` is ignored. -- The instrumentor patches Agno's run functions and every model class in `agno.models`, so agents created after `tracing.py` runs are traced with no further changes. - Without `enable_genai_semconv`, spans carry only OpenInference attributes (`llm.input_messages.0.message.content`, `llm.token_count.prompt`). -- Maple reads span attributes only; the instrumentor emits no span events or OTLP logs, so nothing else needs enabling. Masked values are replaced with `__REDACTED__` in-process before export. -- Team trace shape (one trace per team run): member runs sit directly under the leader's run, next to (not inside) the `delegate_task_to_member` tool spans; those show as ordinary tool calls on the leader with member id and task as arguments. With `team.arun()` in `coordinate` mode, members called in one step run concurrently and their spans overlap; sync `team.run()` runs them sequentially. +- Teams: `delegate_task_to_member` tool spans show as ordinary tool calls on the leader with member id and task as arguments. With `team.arun()` in `coordinate` mode, members called in one step run concurrently and their spans overlap; sync `team.run()` runs them sequentially. - Team context leak is upstream issue agno#5573. "Failed to detach context" in the logs after a streamed team run is agno#5208: log noise, spans still export. - Human-in-the-loop: the paused run (`.run`) and the resumed run (`.continue_run`) are two traces and two turns in one session; the approved tool call appears once, in the second. Instrumentor <1.0.8 doesn't wrap `continue_run()`, so resumed runs appear as loose model/tool calls with no session. - Tokens: every model span has input/output tokens plus cache read/write when the provider reports them; streamed runs record usage from the final chunk. The run span has no tokens, so nothing is double counted. Reasoning tokens aren't broken out. Maple links tool results to calls through the message history since tool spans lack `gen_ai.tool.call.id`, and can't show streaming latency (no TTFT). - Cost: OpenRouter's price is recorded as `llm.cost.total` (USD, instrumentor >=1.0.10). Providers called directly (OpenAI, Anthropic) return no price, so sessions are unpriced. -- A tool that raises: Agno catches it and hands the message to the model; the tool span is `ERROR`, the run span stays OK. Maple groups repeated failures by the exception message. +- A tool that raises: Agno catches it and hands the message to the model; the tool span is `ERROR`, the run span stays OK. ## Do not -- Do not rely on `setup_tracing()` or `AgentOS(tracing=True)` to reach Maple; they only write to the database. -- Do not omit `TraceConfig(enable_genai_semconv=True)` / `OPENINFERENCE_ENABLE_GENAI_SEMCONV=true`. -- Do not call `AgnoInstrumentor().instrument()` twice or create a second `TracerProvider`. -- Do not stack another LLM instrumentor (OpenAI/LiteLLM OpenInference, OpenLIT, `auto_instrument=True`) on the same calls. -- Do not run agents without `session_id=` in a server; do not generate a fresh id per request. -- Do not put a different `session_id` on team members or resumed runs than on the conversation. - Do not stamp `maple_ai.session.id` on Agno spans: it re-vendors them and loses decoding. Agno's own `session.id` is what Maple reads. -- Do not use a console/stdout exporter in production, and do not use `SimpleSpanProcessor` in servers (it exports synchronously on the request path). -- Do not skip the flush in scripts, notebooks, CLIs and serverless. -- Do not run an agent after a team run in the same thread/task without isolating the team run (Step 5). -- Do not expect `gen_ai.response.id`, `gen_ai.tool.call.id` on tool spans, TTFT or reasoning-token attributes from this instrumentor; they are not emitted and nothing needs fixing. -- Do not set `OTEL_SDK_DISABLED=true` (it turns off all tracing). `AGNO_TELEMETRY=false` only stops Agno's product analytics and is safe. diff --git a/skills/maple-agent-tracing-claude-agent-sdk/SKILL.md b/skills/maple-agent-tracing-claude-agent-sdk/SKILL.md index 0b6c5d638..fc5404088 100644 --- a/skills/maple-agent-tracing-claude-agent-sdk/SKILL.md +++ b/skills/maple-agent-tracing-claude-agent-sdk/SKILL.md @@ -5,13 +5,11 @@ description: "Trace Claude Agent SDK agents (TypeScript and Python) and Claude C # Maple agent tracing: Claude Agent SDK and Claude Code -Human guide with the reasoning: https://maple.dev/docs/agent-tracing/claude-agent-sdk - ## Goal One conversation = one Maple Agent Session, one turn per user message, with the user prompts, every model call (model, tokens, TTFT), every tool call (name, args, result, failures). -How it works: the Agent SDK emits nothing itself. `query()` spawns the Claude Code CLI, which has OpenTelemetry built in and exports spans `claude_code.interaction` (turn), `claude_code.llm_request` (model call), `claude_code.tool` (tool call), with phase children `claude_code.tool.blocked_on_user` / `claude_code.tool.execution`. All configuration is environment variables for that child process. Maple detects the `com.anthropic.claude_code*` scopes (spans come from `com.anthropic.claude_code.tracing`), groups by the `session.id` span attribute, and restates the spans as `gen_ai.*` at ingest. No instrumentation package, no TracerProvider. +How it works: the Agent SDK emits nothing itself. `query()` spawns the Claude Code CLI, which has OpenTelemetry built in and exports spans `claude_code.interaction` (turn), `claude_code.llm_request` (model call), `claude_code.tool` (tool call), with phase children `claude_code.tool.blocked_on_user` / `claude_code.tool.execution`. All configuration is environment variables for that child process. No instrumentation package, no TracerProvider. Known gaps (tell the user, don't try to fix): assistant reply text and cost are only on OTLP log events, which Maple's session views don't read, so transcripts have no assistant text and sessions show "unpriced"; no `gen_ai.agent.name`, so sub-agents get no separate lanes; tool arguments shown only for Bash (command) and Read/Edit/Write (file path). @@ -234,15 +232,5 @@ If spans exist but the turn nests under an unrelated trace, an inherited `TRACEP ## Do not -- Do not omit `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1`: zero spans without it (metrics/logs still flow, which hides the problem). -- Do not leave `OTEL_EXPORTER_OTLP_PROTOCOL` unset or use `grpc`: Claude Code has no default; Maple ingest is OTLP/HTTP. -- Do not set any exporter to `console` in SDK apps: stdout is the SDK message channel. -- Do not pass a TS `env` without `...process.env`. -- Do not pass an inherited `TRACEPARENT`/`TRACESTATE` to the CLI (Claude Code's Bash tool and CI set them). -- Do not call `query()` per message without `resume`. -- Do not put telemetry vars in a repo's `.claude/settings.json`. -- Do not assume `options.env` wins: `env` in loaded settings files overrides it (Step 0). -- Do not add a second instrumentation (OpenInference/Langfuse/LangSmith hooks) on top of the CLI's spans. -- Do not use `OTEL_METRICS_INCLUDE_SESSION_ID=false`, `forkSession` on normal turns, or detailed beta tracing. -- Do not promise cost or assistant replies in Maple's session views; they are in Logs (`claude_code.api_request` `cost_usd`, `claude_code.assistant_response`) and the `claude_code.cost.usage` metric. -- Do not print or commit `maple_sk_` keys or model API keys. +- Do not omit `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1`: zero spans, while metrics/logs still flow and hide it. +- Do not leave `OTEL_EXPORTER_OTLP_PROTOCOL` unset or `grpc`: Claude Code has no default; Maple ingest is OTLP/HTTP. diff --git a/skills/maple-agent-tracing-cloudflare-agents/SKILL.md b/skills/maple-agent-tracing-cloudflare-agents/SKILL.md index 70c21847a..4ef1b9801 100644 --- a/skills/maple-agent-tracing-cloudflare-agents/SKILL.md +++ b/skills/maple-agent-tracing-cloudflare-agents/SKILL.md @@ -5,16 +5,10 @@ description: "Trace Cloudflare Agents SDK agents (AIChatAgent, Agent on Durable # Maple agent tracing: Cloudflare Agents SDK -Human guide: https://maple.dev/docs/agent-tracing/cloudflare-agents - ## Goal One chat (one agent instance) = one Maple Agent Session, one turn per user message, with the transcript, every model call (model, tokens), every tool call (name, args, result, failures), and a lane per sub-agent. -How it works: the Agents SDK does not emit GenAI spans itself. Model calls go through the Vercel AI SDK, whose `@ai-sdk/otel` integration emits `invoke_agent ` → `step ` → `chat ` + `execute_tool ` on tracer scope `gen_ai`. Maple detects them as **Vercel AI SDK** (by the `gen_ai` tracer scope) and groups sessions by the conversation id you pass in `runtimeContext` (recorded as `ai.settings.context.conversationId` once `runtimeContext: true` is on). Workers can't run `@opentelemetry/sdk-node`, so you build a `BasicTracerProvider` with the OTLP HTTP exporter (its browser/worker build posts OTLP JSON with `fetch`) and call `forceFlush()` at the end of each turn. - -Verified end to end (wrangler dev, workerd 1.20260926.1, a local OTLP receiver, `MockLanguageModelV4` from `ai/test`) with: `agents` 0.24.0, `@cloudflare/ai-chat` 0.12.0, `ai` 7.0.122, `@ai-sdk/otel` 1.0.122, `wrangler` 4.143.0, `@opentelemetry/sdk-trace-base` / `resources` / `context-async-hooks` 2.11.0, `@opentelemetry/exporter-trace-otlp-http` 0.222.0, `@opentelemetry/api` 1.9.1. Both `AIChatAgent` over its WebSocket protocol (2 turns x 2 chats, streamed, tool call, sub-agent) and a plain `Agent.onRequest` with `generateText`. - Known gaps (tell the user, don't try to fix): cost shows as "unpriced" (the AI SDK emits no cost). These traces are separate from Cloudflare's native Workers traces (different trace ids); that's expected. ## Step 0: Detect @@ -169,11 +163,4 @@ Then against Maple: the `wrangler dev` output shows no export errors (the diag l ## Do not -- Do not use `@opentelemetry/sdk-node` / `NodeSDK` / `@vercel/otel` in a Worker. -- Do not rely on Cloudflare's native trace destinations or `cloudflare:workers` `tracing` for the agent spans; they don't carry AI SDK spans. -- Do not skip the per-turn flush. -- Do not call `registerTelemetry` twice or register `LegacyOpenTelemetry` next to `OpenTelemetry` (duplicate spans, double tokens). -- Do not use a per-request id, `ctx.id.toString()` of a shared DO, or a module constant as the conversation id. -- Do not stamp `maple_ai.session.id` on AI SDK spans: it re-vendors them and loses AI SDK decoding. - Do not add a second AI SDK tracer that also exports to Maple (Langfuse/Braintrust/Sentry AI integrations): every model call gets recorded twice. -- Do not put the ingest key in `vars` in the Wrangler config or in client code; use a secret. diff --git a/skills/maple-agent-tracing-crewai/SKILL.md b/skills/maple-agent-tracing-crewai/SKILL.md index 130be8dae..9762dcf30 100644 --- a/skills/maple-agent-tracing-crewai/SKILL.md +++ b/skills/maple-agent-tracing-crewai/SKILL.md @@ -5,9 +5,9 @@ description: "Trace CrewAI crews and flows with Maple: OpenInference CrewAI inst # Maple agent tracing: CrewAI -Goal: every conversation the app runs through CrewAI shows up in Maple **Agent Sessions** as ONE session, with the transcript, each model call (model, tokens), each tool call (name, result, failure) and one lane per agent role. Reasoning and background for every step: https://maple.dev/docs/agent-tracing/crewai +Goal: every conversation the app runs through CrewAI shows up in Maple **Agent Sessions** as ONE session, with the transcript, each model call (model, tokens), each tool call (name, result, failure) and one lane per agent role. -CrewAI exports nothing to your backend. Its built-in telemetry is anonymous analytics to crewai.com on a private provider; its OTel export is AMP-only. All spans come from OpenInference, and the defaults are wrong for Maple in three ways this skill fixes: the CrewAI instrumentor records no model calls (a second, SDK-level instrumentor is required), no session id, and no agent-name attribute. +CrewAI exports nothing to your backend. All spans come from OpenInference, and the defaults are wrong for Maple in three ways this skill fixes: the CrewAI instrumentor records no model calls (a second, SDK-level instrumentor is required), no session id, and no agent-name attribute. ## Step 0: Detect versions and existing OpenTelemetry @@ -209,25 +209,16 @@ Otherwise check in Maple **Agent Sessions** (`https://app.maple.dev/agent-sessio - Cost: unpriced unless models go through LiteLLM (which records `llm.cost.total`). Expected. - No spans with scope `crewai.telemetry`, no `coding_agent` attribute (telemetry is off). -More details (from the human guide, for edge cases): +Edge cases: - Tokens: model spans carry `gen_ai.usage.input_tokens`/`output_tokens` (plus cached and reasoning tokens when reported); crew and agent spans carry none, so nothing double-counts. CrewAI's OpenAI provider always requests `stream_options={"include_usage": True}` when streaming, so streamed calls keep tokens. - Model = the one the provider returned (e.g. `anthropic/claude-haiku-4.5` behind OpenRouter); provider = the SDK used, so every OpenRouter model shows `openai`. - `memory=True` / `planning=True` add real, billed model calls (memory analysis, embeddings, planning agent); they appear in the session. Expected. - The Arize Phoenix CrewAI page still recommends the LiteLLM instrumentor; ignore it for native providers (it records nothing there). - Flow span layout: `.kickoff` root, one `.` span per `@start`/`@listen`/`@router` method, crews and `Agent.kickoff()` nested inside. Conversational flow turns show `.route_conversation` and `.converse_turn` under the kickoff. -- Why the streaming wrapper works: `gen_ai.operation.name=invoke_agent` makes Maple treat it as the turn's agent span, so the two crew kickoff spans under it are one turn. If sessions are split per message: `using_session` missing or id changing. No model spans/tokens: wrong or missing SDK instrumentor. Every call its own trace: `akickoff`. Nothing arrives: exporter endpoint/header wrong, `.env` loaded after `tracing.py`, `OTEL_SDK_DISABLED=true`, or process exited without flushing. 401 `ingest_unauthorized` / "Invalid ingest key" with a key you trust: keys are region-bound, so it likely belongs to the other region; try the other endpoint. ## Do not -- Do not install only `openinference-instrumentation-crewai`: no model calls, no tokens, no transcript. -- Do not add the LiteLLM instrumentor for `openai/`/`openrouter/`/`anthropic/`/`gemini/` models (CrewAI 1.x calls those SDKs natively; LiteLLM records nothing). Do not stack two model-layer instrumentors or `litellm.callbacks=["otel"]` on the same calls (duplicate spans). -- Do not use `OTEL_SDK_DISABLED=true` to silence CrewAI telemetry. -- Do not use `crewai.telemetry`, `share_crew`, or `CREWAI_TRACING_ENABLED=true` as the Maple pipeline. -- Do not use `crew_id`, `crew_key`, `task_id` or a fresh UUID per request as the session id. -- Do not call `akickoff()` on traced crews. -- Do not pass `OTLPSpanExporter(endpoint="https://ingest.maple.dev")` without `/v1/traces`. -- Do not create a second `TracerProvider` when one exists, and do not call `instrument()` twice. -- Do not wrap tools in try/except that returns error strings; let them raise. +- Do not stack two model-layer instrumentors (or `litellm.callbacks=["otel"]`) on the same calls: duplicate model spans and tokens. diff --git a/skills/maple-agent-tracing-dspy/SKILL.md b/skills/maple-agent-tracing-dspy/SKILL.md index 18554af9f..7a07879ec 100644 --- a/skills/maple-agent-tracing-dspy/SKILL.md +++ b/skills/maple-agent-tracing-dspy/SKILL.md @@ -5,9 +5,9 @@ description: "Trace DSPy programs and ReAct agents with Maple: OpenInference DSP # Maple agent tracing: DSPy -Goal: every conversation with the user's DSPy program shows up in Maple **Agent Sessions** as one session, with one turn per call to the program, a transcript, model calls with tokens and cost, tool calls with arguments, results and failures, and one lane per worker module. Human guide with the reasoning: https://maple.dev/docs/agent-tracing/dspy +Goal: every conversation with the user's DSPy program shows up in Maple **Agent Sessions** as one session, with one turn per call to the program, a transcript, model calls with tokens and cost, tool calls with arguments, results and failures, and one lane per worker module. -DSPy emits nothing by itself. Spans come from `openinference-instrumentation-dspy`. It records no tokens, no tool names, no agent spans and no session id; the steps below add all four. Do not skip any step. +Spans come from `openinference-instrumentation-dspy`. It records no tokens, no tool names, no agent spans and no session id; the steps below add all four. Do not skip any step. ## Step 0: Detect versions and existing setup @@ -95,9 +95,6 @@ def _message(role, values): class MapleCallback(BaseCallback): - """Adds what Maple reads and the OpenInference DSPy instrumentor leaves out: - an agent span per program, tool names and arguments, tokens and cost.""" - def __init__(self): self._agents = set() self._lms = {} @@ -248,30 +245,25 @@ Then check in Maple **Agent Sessions** (`https://app.maple.dev/agent-sessions`, - Worker modules appear as separate agents/lanes; with `dspy.Parallel`, all workers are in the same trace as the orchestrator, none orphaned. - The process exited cleanly and the last turn is present (flush ran). -401 `ingest_unauthorized` / "Invalid ingest key" with a key you trust: keys are region-bound, so it likely belongs to the other region; try the other endpoint. For other failed checks, see the troubleshooting list in the human guide. +401 `ingest_unauthorized` / "Invalid ingest key" with a key you trust: keys are region-bound, so it likely belongs to the other region; try the other endpoint. ## Known behaviours (expected, nothing to fix; explain them if the user asks) - `OPENINFERENCE_ENABLE_GENAI_SEMCONV=true` set before `instrument()` runs is equivalent to `TraceConfig(enable_genai_semconv=True)`. The GenAI dual-write never overwrites a key already set, so the callback's values win. - Each `LM.__call__` sits under `Predict.forward`, `Predict(StringSignature).forward` and `ChatAdapter.__call__`; those are DSPy steps, not extra model calls. -- A `dspy.History` input is expanded into earlier user/assistant messages, so each model call repeats the conversation so far. - `dspy.ReAct` catches tool exceptions and hands `Execution error in : ...` back to the model. The tool span is `ERROR` and counted as failed; the `ReAct.forward` span and the user's module span stay `OK`, and the program's return value doesn't reveal the failure. - Tool spans have no `gen_ai.tool.call.id`: ReAct asks the model for the next tool as text fields (`next_tool_name`, `next_tool_args`), not through the provider's tool-calling API. - When `ChatAdapter` can't parse a reply, DSPy retries with `JSONAdapter`: one `Predict` span holds two adapter spans, each with its own `LM.__call__`. Both calls happened and both are billed. - Cost is DSPy's estimate from each history entry's `cost`: on the `lm15` engine from DSPy's bundled model metadata, on the LiteLLM engine LiteLLM's `response_cost`. Maple never prices tokens; a model DSPy can't price shows as **unpriced**. - An `LM` with a custom `engine=` reports whatever usage that engine puts on its response. - Anthropic models run on the LiteLLM engine by default (as does `dspy.LM(..., engine="litellm")`); that is where an extra LiteLLM/OpenAI instrumentor would double-count. -- `dspy.Parallel` copies only DSPy's settings into its worker threads, not the OpenTelemetry context; `ThreadingInstrumentor` is what carries it. Without it, a test fan-out became four traces with most spans orphaned. +- `dspy.Parallel` copies only DSPy's settings into its worker threads, not the OpenTelemetry context; `ThreadingInstrumentor` is what carries it. - Narrower content switches (`OPENINFERENCE_HIDE_INPUT_TEXT`, `OPENINFERENCE_HIDE_OUTPUT_TEXT`, `OPENINFERENCE_HIDE_LLM_INVOCATION_PARAMETERS`) redact parts of each model message; the callback's agent messages follow only `OPENINFERENCE_HIDE_INPUTS` / `OPENINFERENCE_HIDE_OUTPUTS`. - With content hidden the session still shows turns, model and tool calls, tokens and failures, with an empty transcript and no tool arguments or results. ## Do not -- Do not add `openinference-instrumentation-litellm` or `-openai`. DSPy 3.4 runs most models on its own `lm15` engine, where they record nothing; on the LiteLLM engine they add a duplicate model span per call. - Do not name a `dspy.Module` attribute `history`: DSPy appends LM calls to it and a `dspy.History` there crashes every model call (`TypeError: object of type 'History' has no len()`). Pass `dspy.History` as an input field. - Do not set `disable_history=True` or `max_history_size=0`: the callback reads token usage from LM history. -- Do not expect tokens for cache hits (`cache=True` is the default); they cost nothing and are skipped. -- Do not use MLflow autolog or `opentelemetry-instrumentation-genai-dspy` as the Maple path: the first has no session id Maple reads, the second (1.2b0) records no model calls and is not identified as DSPy. -- Do not create a second `TracerProvider` when one exists. -- Do not pass `endpoint=` without `/v1/traces`, or set the env endpoint with it. +- Do not use `opentelemetry-instrumentation-genai-dspy` as the Maple path: 1.2b0 records no model calls and is not identified as DSPy. - Do not trace optimizer runs (`MIPROv2`, `GEPA`, `BootstrapFewShot`) or `dspy.Evaluate` under the production service name; they make hundreds of calls. diff --git a/skills/maple-agent-tracing-genkit/SKILL.md b/skills/maple-agent-tracing-genkit/SKILL.md index 2807b4f84..6a841453f 100644 --- a/skills/maple-agent-tracing-genkit/SKILL.md +++ b/skills/maple-agent-tracing-genkit/SKILL.md @@ -5,14 +5,10 @@ description: "Trace Genkit (TypeScript/Node.js) agents with Maple: export Genkit # Maple agent tracing: Genkit -Human guide with the reasoning: https://maple.dev/docs/agent-tracing/genkit - ## Goal One conversation = one Maple Agent Session, one turn per flow run, with the transcript, every model call (model, tokens) and every tool call (name, args, result, failures). -How it works: Genkit JS traces every action on tracer `genkit-tracer` through the global `@opentelemetry/api`, but only with its own attributes (`genkit:type`, `genkit:name`, `genkit:metadata:subtype`, `genkit:input`, `genkit:output`, `genkit:isRoot`, `genkit:path`, `genkit:state`). It emits no `gen_ai.*` keys (the GenAI semconv instrumentation `genkit_otel` exists for Dart only). Agent Sessions ignores spans without `gen_ai.operation.name`, so a span processor (`GenkitForMaple`) copies the Genkit attributes to GenAI ones in `onEnd` (Genkit only sets input/output when the span ends). Maple groups sessions by `gen_ai.conversation.id`, which the app sets per flow through `setCustomMetadataAttribute("conversationId", id)` (becomes `genkit:metadata:conversationId` on the flow span). - Span tree per flow run: ```text @@ -30,7 +26,7 @@ Known gaps (tell the user, don't try to fix): cost shows as unpriced; `gen_ai.pr ## Step 0: Detect -- `genkit` version in `package.json` / lockfile: need `>= 1.22` (`disableGenkitOTelInitialization` was added in 1.22). Older: upgrade Genkit first. Node.js >= 20. Tested with genkit 1.42.0 on Node.js 26. +- `genkit` version in `package.json` / lockfile: need `>= 1.22` (`disableGenkitOTelInitialization` was added in 1.22). Older: upgrade Genkit first. Node.js >= 20. - Go or Python Genkit: stop; this skill covers TypeScript/JavaScript only. Tell the user. - Existing OpenTelemetry: search for `NodeSDK`, `NodeTracerProvider`, `registerOTel`, `@vercel/otel`, `Sentry.init`, `enableTelemetry(`, `enableFirebaseTelemetry(`, `enableGoogleCloudTelemetry(`, `ENABLE_FIREBASE_MONITORING`. - An SDK/provider already exists: reuse it. Add `GenkitForMaple` and one Maple exporting processor to it. Never start a second SDK. @@ -58,7 +54,7 @@ Use the repo's package manager. `@opentelemetry/api` arrives as a peer; add it e OTEL_SERVICE_NAME=support-agent OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev -OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer +OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " ``` Inlining instead of env: `new OTLPTraceExporter({ url: "https://ingest.maple.dev/v1/traces", headers: { authorization: "Bearer " } })` (the full `/v1/traces` path is needed when passing `url`). @@ -68,7 +64,110 @@ Inlining instead of env: `new OTLPTraceExporter({ url: "https://ingest.maple.dev ## Step 3: The span processor -Create `genkit-for-maple.ts` next to the entry point. Copy it verbatim from the human guide ("Add the span processor"): https://maple.dev/docs/agent-tracing/genkit. It maps: +Create `genkit-for-maple.ts` next to the entry point, verbatim: + +```ts +// genkit-for-maple.ts +import type { ReadableSpan, SpanProcessor } from "@opentelemetry/sdk-trace-base" + +type Part = { + text?: string + reasoning?: string + toolRequest?: { name: string; ref?: string; input?: unknown } + toolResponse?: { name: string; ref?: string; output?: unknown } +} +type Message = { role: string; content: Part[] } + +// Genkit message parts to OpenTelemetry GenAI parts. Media parts are left out. +function toPart({ text, reasoning, toolRequest, toolResponse }: Part) { + if (text !== undefined) return { type: "text", content: text } + if (reasoning !== undefined) return { type: "reasoning", content: reasoning } + if (toolRequest) { + return { type: "tool_call", id: toolRequest.ref, name: toolRequest.name, arguments: toolRequest.input } + } + if (toolResponse) return { type: "tool_call_response", id: toolResponse.ref, response: toolResponse.output } + return undefined +} + +const toParts = (content: Part[]) => content.map(toPart).filter((part) => part !== undefined) + +function toMessages(messages: Message[]) { + return messages.map((m) => ({ role: m.role === "model" ? "assistant" : m.role, parts: toParts(m.content) })) +} + +/** Adds the gen_ai.* attributes Maple reads to Genkit's flow, model and tool spans. */ +export class GenkitForMaple implements SpanProcessor { + onStart() {} + + onEnd(span: ReadableSpan) { + const attrs = span.attributes + const json = (key: string) => { + const value = attrs[key] + return typeof value === "string" ? JSON.parse(value) : undefined + } + const name = String(attrs["genkit:name"]) + + switch (attrs["genkit:metadata:subtype"]) { + case "flow": + case "agent": { + Object.assign(attrs, { + "gen_ai.operation.name": "invoke_agent", + "gen_ai.agent.name": name, + }) + // Set in your flow, or by Genkit for defineAgent() chats + const conversationId = attrs["genkit:metadata:conversationId"] ?? attrs["genkit:metadata:agent:sessionId"] + if (conversationId !== undefined) attrs["gen_ai.conversation.id"] = conversationId + break + } + case "model": { + const input = json("genkit:input") + const output = json("genkit:output") + const [provider, ...model] = name.split("/") + const messages: Message[] = input?.messages ?? [] + const system = messages.filter((m) => m.role === "system").flatMap((m) => toParts(m.content)) + Object.assign(attrs, { + "gen_ai.operation.name": "chat", + "gen_ai.provider.name": provider, + "gen_ai.request.model": model.join("/") || name, + "gen_ai.input.messages": JSON.stringify(toMessages(messages.filter((m) => m.role !== "system"))), + }) + if (system.length > 0) attrs["gen_ai.system_instructions"] = JSON.stringify(system) + if (output?.message) { + attrs["gen_ai.output.messages"] = JSON.stringify( + toMessages([output.message]).map((m) => ({ ...m, finish_reason: output.finishReason })), + ) + } + if (output?.finishReason) attrs["gen_ai.response.finish_reasons"] = [output.finishReason] + if (output?.usage?.inputTokens !== undefined) attrs["gen_ai.usage.input_tokens"] = output.usage.inputTokens + if (output?.usage?.outputTokens !== undefined) attrs["gen_ai.usage.output_tokens"] = output.usage.outputTokens + break + } + case "tool": { + Object.assign(attrs, { + "gen_ai.operation.name": "execute_tool", + "gen_ai.tool.name": name, + "gen_ai.tool.call.arguments": attrs["genkit:input"] ?? "{}", + }) + const result = json("genkit:output") + if (result !== undefined) { + attrs["gen_ai.tool.call.result"] = typeof result === "string" ? result : JSON.stringify(result) + } + break + } + } + } + + forceFlush() { + return Promise.resolve() + } + + shutdown() { + return Promise.resolve() + } +} +``` + +It maps: | Genkit span (`genkit:metadata:subtype`) | Adds | | --- | --- | @@ -87,7 +186,7 @@ Notes: ## Step 4: Start OpenTelemetry -Create `instrumentation.ts` (copy from the guide, "Start OpenTelemetry"): +Create `instrumentation.ts`: ```ts import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto" @@ -141,7 +240,7 @@ setCustomMetadataAttribute("conversationId", chatId) - Streaming (`ai.generateStream`, `flow.stream()`): model spans end with the full output and usage; read the stream to the end before flushing. - Flush: - Scripts/CLIs: `await sdk.shutdown().catch((err) => console.error("telemetry flush failed", err))` in a `finally`. `shutdown()` rejects when an export failed; the `.catch` keeps a Maple outage from crashing the app. - - Long-running servers (Cloud Run, plain Node): `process.on("SIGTERM", () => sdk.shutdown().catch((err) => console.error("telemetry flush failed", err)))`; nothing per request. Genkit's own SDK did this; Maple's doesn't by default. + - Long-running servers (Cloud Run, plain Node): `process.on("SIGTERM", () => sdk.shutdown().catch((err) => console.error("telemetry flush failed", err)))`; nothing per request. - Serverless handlers you control: `await spanProcessor.forceFlush()` after the flow returns (the flow span ends when the flow returns, so flushing inside the flow misses it). - Cloud Functions for Firebase `onCallGenkit(flow)`: you can't hook after the flow; the instance may be throttled after the response, so the last batch can be delayed or lost. Tell the user; if it matters, wrap with `onCall` and call the flow then `forceFlush()` before returning. Untested. @@ -163,12 +262,4 @@ Without Maple access: the run exits with no export errors on stderr (`OTLPExport ## Do not -- Do not pass Maple's exporter or processors to Genkit's `enableTelemetry()`; it crashes with current OpenTelemetry packages. -- Do not skip `GenkitForMaple`: raw Genkit spans reach Maple Traces but never Agent Sessions. -- Do not start a second OpenTelemetry SDK next to an existing one; add the processors. -- Do not import `instrumentation.ts` after Genkit has run anything. -- Do not call `setCustomMetadataAttribute("conversationId", ...)` inside tools or nested flows, or outside a Genkit action. -- Do not stamp `maple_ai.session.id` on Genkit spans. - Do not add a second GenAI tracer that also exports to Maple (OpenLLMetry/OpenInference Google GenAI or OpenAI instrumentations for the same calls): model calls get recorded twice. -- Do not set attribute length limits. -- Do not use a gRPC exporter; Maple ingest is OTLP over HTTP. diff --git a/skills/maple-agent-tracing-google-adk/SKILL.md b/skills/maple-agent-tracing-google-adk/SKILL.md index c68b95ca0..502101a73 100644 --- a/skills/maple-agent-tracing-google-adk/SKILL.md +++ b/skills/maple-agent-tracing-google-adk/SKILL.md @@ -5,7 +5,7 @@ description: "Trace Google ADK (Agent Development Kit) agents with Maple, in Pyt # Maple agent tracing: Google ADK (Python and TypeScript) -Goal: one conversation = one ADK session id = one Maple Agent Session, with the transcript (user, assistant, tool calls and results), model calls, tool calls with arguments/results, failures, and tokens. Reasoning for every step: https://maple.dev/docs/agent-tracing/google-adk +Goal: one conversation = one ADK session id = one Maple Agent Session, with the transcript (user, assistant, tool calls and results), model calls, tool calls with arguments/results, failures, and tokens. ADK emits its own OTel spans (scope `gcp.vertex.agent`): `invocation` > `invoke_agent {agent}` > `call_llm` > `generate_content {model}`, plus `execute_tool {tool}`. No instrumentation package is needed. You add: a tracer provider (Runner apps only), the env vars below, one plugin, one span processor. @@ -24,7 +24,7 @@ ADK emits its own OTel spans (scope `gcp.vertex.agent`): `invocation` > `invoke_ ## Step 1: Key and region - US: `https://ingest.maple.dev`. EU: `https://ingest.eu.maple.dev`. -- Header: `Authorization=Bearer `. In `OTEL_EXPORTER_OTLP_HEADERS` write the space as `%20`: `Authorization=Bearer%20`. +- Header: `Authorization=Bearer `. - Key given in the prompt: use it. No key: use the literal `MAPLE_TEST` (ingest accepts and discards it) and tell the user to replace it with a key from Settings → Ingestion. - Never put a private `maple_sk_` key in browser code. - Follow the repo's secret/env convention (`.env`, settings module, deployment env). If there is none, inline values are acceptable: ingest keys are write-only. @@ -43,7 +43,7 @@ Env vars (all runtimes, including `adk web`/`api_server`): ```bash OTEL_SERVICE_NAME= OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev -OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20 +OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false @@ -183,19 +183,11 @@ Tell the user about the known gaps: cost is unpriced; the model is the requested - Tokens: `generate_content` carries `gen_ai.usage.input_tokens` / `output_tokens`, plus `gen_ai.usage.cache_read.input_tokens` and `gen_ai.usage.reasoning.output_tokens` when reported (cached inside input, thinking inside output). `call_llm` repeats the same usage; Maple counts it on `generate_content` only, so totals and LLM call count are correct. Streamed turns (`StreamingMode.SSE`) report usage; `LiteLlm` requests it via `stream_options.include_usage`. - Cost: LiteLLM computes a cost but it never reaches ADK's spans; sessions are unpriced. -- `AgentTool` detail: Maple keeps one `gen_ai.conversation.id` per trace and picks the larger of the two, which is why the turn can move to another session. ADK's API docs also prefer `mode="single_turn"`. +- `AgentTool` detail: Maple keeps one `gen_ai.conversation.id` per trace and picks the larger of the two, which is why the turn can move to another session. - The approval `run_async()` is its own turn, labeled with the original request. - With the settings in this skill, `generate_content` spans carry no provider attribute; doesn't affect grouping, tokens or transcript. -- 401 from the exporter (`ingest_unauthorized`, "Invalid ingest key"): the header must be `Authorization=Bearer%20`. With a key you trust, it usually belongs to the other region (keys are region-bound): try the other endpoint. +- 401 from the exporter (`ingest_unauthorized`, "Invalid ingest key"): with a key you trust, it usually belongs to the other region (keys are region-bound): try the other endpoint. ## Do not -- Do not rely on `OTEL_EXPORTER_OTLP_*` env alone with a plain `Runner`: nothing is exported. -- Do not set `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true` (log records only) or leave out `OTEL_SEMCONV_STABILITY_OPT_IN`: the transcript stays empty. -- Do not create a second `TracerProvider` when one exists; only the first global one wins. -- Do not add `litellm.callbacks=["otel"]`, OpenInference ADK/LiteLLM/OpenAI instrumentors, or the OpenAI OTel instrumentor alongside ADK's spans: doubled model calls and tokens. -- Do not create a new ADK session per request, or share one session id across users. -- Do not use `AgentTool` for sub-agents when tracing matters; use `mode="single_turn"` sub-agents. - Do not drop or filter `call_llm` spans: `generate_content` would lose its parent. -- Do not use the gRPC OTLP exporter or pin OTel packages above ADK's cap. -- Do not skip `force_flush()`/`shutdown()` in short-lived processes. diff --git a/skills/maple-agent-tracing-google-adk/references/typescript.md b/skills/maple-agent-tracing-google-adk/references/typescript.md index c544e6fcc..dd67c1941 100644 --- a/skills/maple-agent-tracing-google-adk/references/typescript.md +++ b/skills/maple-agent-tracing-google-adk/references/typescript.md @@ -1,8 +1,6 @@ # Google ADK for TypeScript (`@google/adk`) -Human guide (TypeScript tab): https://maple.dev/docs/agent-tracing/google-adk - -Tested with `@google/adk` 2.1.0 (`@google/genai` 2.24.0), `@opentelemetry/sdk-node` 0.222.0 (trace SDK 2.11.0), `@opentelemetry/exporter-trace-otlp-proto` 0.222.0, `zod` 4.6.5, Node.js 26.0, TypeScript 7.0 `strict`. Gemini was mocked with a local server (`GOOGLE_GEMINI_BASE_URL`), spans read from a local OTLP receiver: two turns in one session with a tool call, streaming (`StreamingMode.SSE`), a throwing tool, an unknown tool, and two parallel tool calls. +Tested with `@google/adk` 2.1.0 (`@google/genai` 2.24.0), `@opentelemetry/sdk-node` 0.222.0 (trace SDK 2.11.0), `@opentelemetry/exporter-trace-otlp-proto` 0.222.0, `zod` 4.6.5, Node.js 26.0, TypeScript 7.0 `strict`. ## What ADK for TypeScript emits, and what is missing @@ -18,7 +16,7 @@ invocation └─ call_llm ``` -Gaps against what Maple reads (compare the Python side, which has a `generate_content` span with GenAI messages): +Gaps against what Maple reads: - `call_llm` has no `gen_ai.operation.name`. - No `gen_ai.input.messages` / `gen_ai.output.messages` / `gen_ai.system_instructions`. Content exists only as Gemini-shaped JSON in `gcp.vertex.agent.llm_request` (`{model, contents, config: {systemInstruction, tools}}`) and `gcp.vertex.agent.llm_response` (`{content, usageMetadata, finishReason}`), which Maple does not read. There is no `OTEL_SEMCONV_STABILITY_OPT_IN` switch in the TS package. @@ -40,7 +38,7 @@ Why a span processor and not a plugin (the Python fix): ADK-TS runs `beforeToolC ## Step 1: Key and region -Same as SKILL.md Step 1. `OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer "`: the JS exporter accepts a literal space (in quotes) or `%20`; both arrive as `Bearer `. +Same as SKILL.md Step 1. `OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer "`. A 401 `ingest_unauthorized` ("Invalid ingest key") with a key you trust usually means the key belongs to the other region (keys are region-bound): try the other endpoint. @@ -62,7 +60,7 @@ OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " Leave `ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS` unset (default `true`): the processor reads those attributes. Do not set `OTEL_EXPORTER_OTLP_PROTOCOL`; the exporter class decides the protocol. -`instrumentation.ts` (identical to the guide; typechecks under `strict`): +`instrumentation.ts` (typechecks under `strict`): ```ts // instrumentation.ts @@ -222,10 +220,4 @@ Tell the user the known gaps: cost unpriced; model is the requested id (no respo ## Do not -- Do not rely on `new NodeSDK()` with env alone: spans export, but with no transcript and no tool arguments/results. -- Do not set `ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false` unless the user wants no content. -- Do not port the Python plugin (`before_tool_callback` setting span attributes): in TS the callbacks run outside the tool span and would annotate `call_llm` instead. -- Do not start a second `NodeSDK`/provider next to an existing one; add the processor to it. -- Do not create a new ADK session per request, or share one session id across users. -- Do not drop `call_llm` or `invocation` spans. -- Do not skip `sdk.shutdown()` / `forceFlush()` in short-lived processes. +- Do not drop `call_llm` or `invocation` spans in the processor: their children lose their parent. diff --git a/skills/maple-agent-tracing-haystack/SKILL.md b/skills/maple-agent-tracing-haystack/SKILL.md index 3063f9c05..1fef718c3 100644 --- a/skills/maple-agent-tracing-haystack/SKILL.md +++ b/skills/maple-agent-tracing-haystack/SKILL.md @@ -5,9 +5,7 @@ description: "Trace Haystack agents with Maple: installs a small Haystack tracer # Maple agent tracing: Haystack -Goal: one conversation = one Maple Agent Session, with transcript, model calls, tool calls (failures marked), tokens and (on OpenRouter) cost. Reasoning and details: https://maple.dev/docs/agent-tracing/haystack - -Haystack's `OpenTelemetryTracer` alone gives Maple structure only (model, tokens, content live in `haystack.*` JSON blobs Maple doesn't read; failed tools end `Unset`; no conversation id). The fix is `MapleHaystackTracer`, a subclass that writes `gen_ai.*` attributes on the live spans. +Goal: one conversation = one Maple Agent Session, with transcript, model calls, tool calls (failures marked), tokens and (on OpenRouter) cost. ## Step 0: Detect @@ -181,7 +179,7 @@ Env: ```bash OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev # EU: https://ingest.eu.maple.dev; /v1/traces is appended -OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer +OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf ``` @@ -250,14 +248,7 @@ Run one real conversation (≥2 turns, one tool call). No scriptable entry point Tell the user: known gaps are no `gen_ai.provider.name`, no `gen_ai.response.id`, no tool call id on tool spans, and cost only on OpenRouter. -## Reference: span mapping and expected trace - -| Haystack span | What `MapleHaystackTracer` adds | -| --- | --- | -| `haystack.agent.run` | `invoke_agent`, agent name from the pipeline component or `AgentTool` that runs it | -| `haystack.agent.step.llm`, and any `*ChatGenerator` component | `chat`: model, finish reason, input/output/cache/reasoning tokens, cost, messages | -| `haystack.agent.step.tool` | `execute_tool`: tool name, arguments, result, and `Error` status when the tool failed | -| every span | `gen_ai.conversation.id` inside a `conversation()` block | +## Expected trace ```text haystack.pipeline.run @@ -272,23 +263,13 @@ haystack.pipeline.run ## Known behaviours (expected; explain if the user asks) -- Why not the ready-made options: plain `OpenTelemetryTracer` gives structure only (model, tokens, transcript, tool failures stay in `haystack.*` blobs; no session id). `openinference-instrumentation-haystack` gives model and tokens on generator spans but no tool spans (the Agent is one opaque chain). OpenLLMetry's `opentelemetry-instrumentation-haystack` covers only `Pipeline.run`, `OpenAIGenerator` and `OpenAIChatGenerator`, no Agent steps, tool spans or tokens, and writes content as indexed `gen_ai.prompt.N.*` keys Maple doesn't read. -- Maple also recognizes Haystack's span names, but the `"haystack"` scope covers spans added in newer Haystack releases too. - Haystack looks up the active tracer on every span, so there is no import-order trap for `enable_tracing()`. Without this tracer, `HAYSTACK_CONTENT_TRACING_ENABLED` is read once at the first `import haystack`; setting it later silently records nothing. - Tools of one step run in parallel threads; their spans sit side by side under the step. - Haystack wraps a raised tool exception in `ToolInvocationError`, feeds the text back to the model, writes `{"error": "..."}` as the tool output and leaves the span `Unset`. The tracer sets `Error` regardless of `raise_on_tool_invocation_failure`. - An Agent inside a `PipelineTool` is named after its component in the inner pipeline. -- The Agent itself reports no usage, so there is no double counting: each model call counts once on its `haystack.agent.step.llm` span. - Approval gates (`ConfirmationHook` at `before_tool`) stay in the same run and session. A rejected call never reaches the tool, so it has no tool span; the model sees the rejection as a tool result in the transcript. An Agent with hooks also gets a `haystack.agent.hook` span before its tool calls; like `haystack.agent.step`, it carries no model or tool attributes and isn't counted. - `haystack.pipeline.input_data` on the root span is a plain tag Haystack's own content switch never gated; `content=False` drops it. To redact rather than drop message content, filter values in `_messages()`. ## Do not -- Don't rely on `OpenTelemetryTracer`/`OpenTelemetryConnector` alone: Maple gets no model, tokens, transcript or tool failures. -- Don't add `openinference-instrumentation-haystack` or OpenLLMetry's Haystack instrumentor alongside (double model calls; OpenInference has no tool spans). -- Don't create a second `TracerProvider`; don't rename the tracer from `"haystack"`. -- Don't mint a conversation id per request or share one across users. -- Don't set `session.id` or `maple_ai.session.id` for Haystack; `gen_ai.conversation.id` via `conversation()` is the key. -- Don't expect Haystack 2.x auto-tracing: 3.x needs `enable_tracing(...)`. -- Don't use a console exporter in production or forget the flush in short-lived processes. -- Don't modify `maple_haystack.py` beyond extending usage keys or redaction. +- Don't set `maple_ai.session.id` for Haystack; `gen_ai.conversation.id` via `conversation()` is the key. diff --git a/skills/maple-agent-tracing-langchain/SKILL.md b/skills/maple-agent-tracing-langchain/SKILL.md index d218ea2ca..cdd8f0547 100644 --- a/skills/maple-agent-tracing-langchain/SKILL.md +++ b/skills/maple-agent-tracing-langchain/SKILL.md @@ -7,9 +7,7 @@ description: "Trace LangChain and LangGraph agents (Python, and LangChain.js / L Goal: every conversation = one Maple Agent Session. Each `invoke()`/`stream()` = one turn (one trace) with a readable transcript, chat model spans with tokens, tool spans with names/results, failed tools marked failed, sub-agents in their own lanes. -Human guide with the reasoning: https://maple.dev/docs/agent-tracing/langchain - -Mechanism: `openinference-instrumentation-langchain` (scope `openinference.instrumentation.langchain`). Maple groups sessions by the run's `thread_id` and reads transcript, tokens and tool calls from its spans. `TraceConfig(enable_genai_semconv=True)` is recommended: it also emits standard GenAI attributes (`gen_ai.conversation.id` from run metadata `session_id` > `conversation_id` > `thread_id`, `gen_ai.input/output.messages` in `{role, parts}` form). This beats LangSmith's OTel export for Maple: readable transcript, interrupts not marked ERROR, no middleware noise spans, normal flush. +Mechanism: `openinference-instrumentation-langchain` (scope `openinference.instrumentation.langchain`). Maple groups sessions by the run's `thread_id` and reads transcript, tokens and tool calls from its spans. `TraceConfig(enable_genai_semconv=True)` is recommended: it also emits standard GenAI attributes (`gen_ai.conversation.id` from run metadata `session_id` > `conversation_id` > `thread_id`, `gen_ai.input/output.messages` in `{role, parts}` form). **TypeScript / JavaScript (LangChain.js, LangGraph.js):** the JS instrumentor has no GenAI dual-write, so the setup adds a small span processor. Do Step 1 below for the key and region, then follow [references/typescript.md](references/typescript.md) instead of Steps 2-7. A repo with both Python and TS agents gets both setups. @@ -41,7 +39,7 @@ Env vars (`OTLPSpanExporter()` with no args reads them and appends `/v1/traces`) OTEL_SERVICE_NAME= OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name= OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev -OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer +OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " ``` If you pass `OTLPSpanExporter(endpoint=...)` in code, it must end in `/v1/traces` (used verbatim). @@ -98,7 +96,6 @@ LangChainInstrumentor().instrument( ) ``` -- `TraceConfig(enable_genai_semconv=True)`: recommended (emits standard GenAI attributes). - Import `tracing` first in the entry point (app module, `main.py`, worker, LangGraph Server graph module). It must run before the first `invoke()`. - The app loads `.env` (`load_dotenv()`, `--env-file`): call `load_dotenv()` at the top of `tracing.py`, before the provider is built. Otherwise the exporter silently targets `localhost:4318` with no key. - Fill `AGENT_NAMES` with every agent's `name=` from Step 0.3. Give unnamed `create_agent(...)` calls a `name=` (default graph name is `LangGraph`). @@ -195,7 +192,7 @@ Run one real conversation: 2+ messages with the same id, at least one tool call, Known gaps (not setup bugs, don't try to fix): tool spans have no `gen_ai.tool.call.id`; chat model spans have no `gen_ai.response.id`. -More details (from the human guide, for edge cases): +More details (edge cases): - `OPENINFERENCE_ENABLE_GENAI_SEMCONV=true` is equivalent to `enable_genai_semconv=True`, only if set before `TraceConfig` is built. The dual-write happens at span end and never overwrites a key already set (so `AgentSpans` values win). - Why `AgentSpans`: the instrumentor marks a span AGENT only when its name contains "agent" (`name="support_agent"` yes, `name="assistant"` no) and never sets `gen_ai.agent.name`. Spans with no operation are classified by name, so the `tools` node counts as a tool call unless marked `invoke_workflow`. @@ -205,18 +202,5 @@ More details (from the human guide, for edge cases): - HITL: the pause ends the turn's trace; the resume is a new trace, so an approved action shows as two turns in one session (shared `thread_id` is what joins them). - LangGraph Server verified with `langgraph dev` (langgraph-api 0.10.3). - LangSmith OTel export (`LANGSMITH_OTEL_ENABLED` + `LANGSMITH_OTEL_ONLY`, tested langsmith 0.14.1): Maple labels it "LangChain" and reads `langsmith.metadata.thread_id`, but prompts/completions arrive as byte attributes (hex, unreadable transcript, no turn labels), interrupts are marked ERROR, agent names only in `langsmith.metadata.lc_agent_name` (unread; LangSmith sets `gen_ai.operation.name` after start so a start-time processor can't fix it), and flush needs `wait_for_all_tracers()` (`langchain_core.tracers.langchain`) then `provider.force_flush()`. Don't recommend it. -- LangChain.js/LangGraph.js: covered in [references/typescript.md](references/typescript.md) (OpenInference JS + a `GenAiSpans` processor, since the JS instrumentor has no GenAI dual-write). Without Maple access, both must hold: the run exits with no export errors on stderr (`Failed to export`, 401 lines), AND a temporary `SimpleSpanProcessor(ConsoleSpanExporter())` shows the agent, chat model and tool spans with `gen_ai.conversation.id` identical on every span of every turn. Silence alone proves nothing (no spans is silent too). With the Maple MCP: `list_agent_sessions` with `search=` returns one row. - -## Do not - -- Do not also enable LangSmith's OTel export (`LANGSMITH_OTEL_ENABLED`, `LANGSMITH_OTEL_ONLY`, `LANGSMITH_TRACING_MODE=otel`) or a provider-level instrumentor (`openinference-instrumentation-openai`/`-anthropic`, OpenLLMetry): duplicate spans and doubled tokens. -- Do not create a second `TracerProvider` when one exists. -- Do not generate a new `thread_id` per request, and do not forget it on `stream()` and `Command(resume=...)` calls. -- Do not rely on `configurable` for plain chains; use `metadata`. -- Do not forget `stream_usage=True` on `ChatOpenAI` with a custom `base_url`. -- Do not name tools with "agent" in them. -- Do not add `maple_ai.session.id` attributes (they re-vendor the span). -- Do not promise cost in Maple; do not add token pricing code. -- Do not print or commit real keys beyond the repo's convention. diff --git a/skills/maple-agent-tracing-langchain/references/typescript.md b/skills/maple-agent-tracing-langchain/references/typescript.md index d093cf3ea..69bdc1ce1 100644 --- a/skills/maple-agent-tracing-langchain/references/typescript.md +++ b/skills/maple-agent-tracing-langchain/references/typescript.md @@ -2,11 +2,9 @@ Follow this instead of Steps 2-7 of SKILL.md when the app is TypeScript/JavaScript. Step 1 (key and region) is shared. -Goal is the same as Python: every conversation = one Maple Agent Session, each `invoke()`/`stream()` = one turn (one trace) with a readable transcript, chat spans with tokens, tool spans with name/arguments/result, failed tools marked failed, sub-agents in their own lanes. - Mechanism: `@arizeai/openinference-instrumentation-langchain` (scope `@arizeai/openinference-instrumentation-langchain`) emits OpenInference attributes only; unlike Python it has NO GenAI dual-write (`enable_genai_semconv` does not exist in JS). Maple groups sessions by its `session.id` and decodes the transcript, but without help finds no agent names and counts the `tools` node as a tool call. The `GenAiSpans` span processor below fixes that: in the SDK's `onEnding` hook (after OpenInference has set its attributes, before the span is frozen) it copies them into `gen_ai.*`. -Tested: langchain 1.5.14, @langchain/core 1.2.13, @langchain/langgraph 1.4.18, @langchain/openai 1.6.0, @arizeai/openinference-instrumentation-langchain 4.1.1, @opentelemetry/sdk-node 0.222.0 (sdk-trace-base 2.11.0), Node.js 26 and Bun 1.3, with `createAgent` + `MemorySaver`, a mock OpenAI-compatible server (invoke and `streamMode: "messages"`), `FakeToolCallingModel`/`FakeListChatModel`, an agent-as-tool sub-agent, a failing tool, plain `prompt.pipe(model)` chains, a 20-turn thread. +Tested: langchain 1.5.14, @langchain/core 1.2.13, @langchain/langgraph 1.4.18, @langchain/openai 1.6.0, @arizeai/openinference-instrumentation-langchain 4.1.1, @opentelemetry/sdk-node 0.222.0 (sdk-trace-base 2.11.0), Node.js 26 and Bun 1.3. ## Step 0: Detect @@ -34,7 +32,7 @@ Env vars (as SKILL.md Step 1): `OTEL_SERVICE_NAME`, `OTEL_RESOURCE_ATTRIBUTES`, ## Step 2: Init -Create `genai-spans.ts` next to the entry point. Copy it unchanged (it is the exact file that was tested): +Create `genai-spans.ts` next to the entry point. Copy it unchanged: ```ts // genai-spans.ts: adds the OpenTelemetry GenAI attributes Maple reads to OpenInference's LangChain spans @@ -266,15 +264,3 @@ Known gaps (not setup bugs): `gen_ai.provider.name` is LangChain's `ls_provider` Why not the alternatives (checked 2026-09): - `@traceloop/instrumentation-langchain` 0.27 (OpenLLMetry) emits `gen_ai.*`, but starts spans without the LangChain parent run, drops tool calls from messages, and has no conversation id. - LangSmith JS OTel export is experimental; Maple labels it "LangChain" but it has the same problems as the Python LangSmith path (see SKILL.md). - -## Do not - -- Do not skip `GenAiSpans`: without it agents have no names and the `tools` node counts as a tool call. -- Do not skip `manuallyInstrument(CallbackManagerModule)`. -- Do not start a second `NodeSDK` / tracer provider when one exists. -- Do not lower `spanLimits.attributeCountLimit` back to the default. -- Do not also enable OpenLLMetry's LangChain instrumentation or LangSmith OTel export: duplicate spans and doubled tokens. -- Do not generate a new `thread_id` per request; do not forget it on `stream()` and `Command` resumes; use `metadata` for plain chains. -- Do not leave `createAgent` calls unnamed. -- Do not add `maple_ai.session.id` attributes. -- Do not promise cost. diff --git a/skills/maple-agent-tracing-litellm/SKILL.md b/skills/maple-agent-tracing-litellm/SKILL.md index f054b1760..e33601dbe 100644 --- a/skills/maple-agent-tracing-litellm/SKILL.md +++ b/skills/maple-agent-tracing-litellm/SKILL.md @@ -7,11 +7,9 @@ description: "Trace LiteLLM agents with Maple: export LiteLLM's v2 OpenTelemetry Goal: every conversation = one Maple Agent Session. Each agent run = one turn (one trace) with transcript, LiteLLM `chat ` spans with tokens, `execute_tool` spans with args/results, failed tools marked failed, sub-agents in their own lanes. -Human guide with the reasoning: https://maple.dev/docs/agent-tracing/litellm - Mechanism: - LiteLLM traces model calls only (scope `litellm`). It has no agent loop, no tool executor, no conversation concept. The app MUST emit `invoke_agent` and `execute_tool` spans itself. -- Use LiteLLM's **v2** OTel logger (`OpenTelemetryV2`). It writes `gen_ai.operation.name=chat`, provider, model, usage, `gen_ai.response.id`, TTFT, `error.type`, and `gen_ai.conversation.id` from `litellm_session_id`. Maple reads `gen_ai.conversation.id` as the session key for LiteLLM. +- Use LiteLLM's **v2** OTel logger (`OpenTelemetryV2`). It sets `gen_ai.conversation.id` (Maple's session key) from `litellm_session_id`. - The default v1 logger (`litellm.callbacks=["otel"]` without v2) is wrong for Maple: op `acompletion`, no conversation id ever, and under an open parent span it writes onto the (ended) parent and the data is dropped. ## Step 0: Detect @@ -42,7 +40,7 @@ Mechanism: ```bash OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev -OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer +OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf ``` @@ -90,7 +88,6 @@ tracer = trace.get_tracer("support-agent") - Import it first in the entry point, before the first model call. - Passing the instance: no `LITELLM_OTEL_V2` env needed, LiteLLM builds no provider/exporter of its own. - If the project appends to `litellm.callbacks` elsewhere (other loggers), append the instance instead of overwriting the list. -- Set a real `service.name`. ## Step 2b: Proxy path (self-hosted LiteLLM Proxy) @@ -201,7 +198,6 @@ async def run_tool(agent: Agent, call) -> str: return output ``` -- Real tool name in `gen_ai.tool.name`, the model's `call.id` in `gen_ai.tool.call.id`. Async tools are awaited inside the span (the `isawaitable` branch). - Tool failure: ERROR status + `error.type` on the tool span; the model still gets an error payload. Where the existing loop swallows exceptions into strings, add the status/`error.type` there (don't change what the model sees unless asked). - LiteLLM marks failed model calls ERROR + `error.type` itself. - Multi-agent: each agent its own `name` (→ `gen_ai.agent.name`, one lane each). Same `conversation_id` to every `run_agent`. @@ -258,17 +254,3 @@ Without Maple access (`MAPLE_TEST`, no MCP): the run must exit with no export er - Nothing arrives from the proxy → the OTLP env vars aren't in the proxy's own environment, so it stays on the default `console` exporter. - 401 `ingest_unauthorized` / "Invalid ingest key" with a key you trust → keys are region-bound, so it likely belongs to the other region; try the other endpoint. - App exports to `localhost:4318` / nothing arrives, no errors → `.env` loaded after `tracing.py` built the provider; load it first. - -## Do not - -- Do not use the v1 logger (`litellm.callbacks=["otel"]` alone) when async is possible: no session key, op `acompletion`, attributes lost under parent spans. -- Do not run LiteLLM 1.103 with OpenTelemetry >= 1.44 (v2 silently disabled). -- Do not register two loggers (v2 instance plus `"otel"` / `LITELLM_OTEL_V2` factory) in one process. -- Do not trace the same call at the proxy and in the app (client instrumentor): double counting. -- Do not rely on sync `litellm.completion()` under v2: no span. -- Do not put `gen_ai.conversation.id` / `maple_ai.session.id` on your own spans in the v2 setup. -- Do not use `event_only` content capture. -- Do not skip the flush in short-lived processes. -- Do not create a second `TracerProvider` when one exists. -- Do not add token pricing code or `gen_ai.usage.cost` on your own spans. -- Do not print or commit real keys beyond the repo's convention. diff --git a/skills/maple-agent-tracing-llamaindex/SKILL.md b/skills/maple-agent-tracing-llamaindex/SKILL.md index ced1c71da..3482c4baa 100644 --- a/skills/maple-agent-tracing-llamaindex/SKILL.md +++ b/skills/maple-agent-tracing-llamaindex/SKILL.md @@ -7,8 +7,6 @@ description: "Trace LlamaIndex agents with Maple: OpenInference LlamaIndex instr Goal: every conversation = one Maple Agent Session. Each `agent.run()` / `workflow.run()` = one turn (one trace) with transcript, one model span per model call with tokens, `FunctionTool.acall` tool spans with results, failed tools marked failed, sub-agents in their own lanes. -Human guide with the reasoning: https://maple.dev/docs/agent-tracing/llamaindex - Mechanism: `openinference-instrumentation-llama-index` (scope `openinference.instrumentation.llama_index`). Maple reads the session (`session.id` from `using_session`), transcript, tokens and tool calls from its spans. `TraceConfig(enable_genai_semconv=True)` is recommended: it also emits standard GenAI attributes (`gen_ai.*`, incl. `gen_ai.conversation.id`). ## Step 0: Detect @@ -41,7 +39,7 @@ Env vars (`OTLPSpanExporter()` reads them and appends `/v1/traces`): OTEL_SERVICE_NAME=support-agent OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev -OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer +OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf ``` @@ -120,7 +118,6 @@ LlamaIndexInstrumentor().instrument( ) ``` -- `TraceConfig(enable_genai_semconv=True)`: recommended (emits standard GenAI attributes). - Import `tracing` first in the entry point (app module, `main.py`, worker). `instrument()` must run before the first `agent.run()`. - The app loads `.env` (`load_dotenv()`, `--env-file`): call `load_dotenv()` at the top of `tracing.py`, before the provider is built. Otherwise the exporter silently targets `localhost:4318` with no key. - Existing provider: skip `TracerProvider()`/`set_tracer_provider`; call `existing.add_span_processor(LlamaIndexForMaple(BatchSpanProcessor(OTLPSpanExporter())))` and pass `tracer_provider=existing`. @@ -197,7 +194,7 @@ llm = OpenAI(model="gpt-4o-mini", additional_kwargs={"stream_options": {"include ## Step 7: Flush -- `BatchSpanProcessor` exports every 5 s; the provider flushes at normal interpreter exit. +- The provider flushes at normal interpreter exit. - Scripts/CLIs/one-shot jobs: `provider.shutdown()` in a `finally` at the end. - Serverless handlers, Celery/RQ tasks, notebooks: `provider.force_flush()` in a `finally` after each run (`from tracing import provider`). - FastAPI/long-running servers: nothing extra; optionally `provider.shutdown()` in the lifespan shutdown. @@ -218,7 +215,7 @@ Run one real conversation: 2+ messages with the same id, at least one tool call, - [ ] Cost shows "unpriced". - [ ] No attribute contains an API key or `Bearer ` token. -More details (from the human guide, for edge cases): +Edge cases: - Native `llama-index-observability-otel` (0.7.0, `LlamaIndexOpenTelemetry`): model settings/prompt only in `LLMChatStartEvent` span events (Maple ignores events); the end event with reply+usage is dropped on streamed calls; two nested `.astream_chat` spans per call; ignores `OTEL_EXPORTER_OTLP_*` and defaults to `ConsoleSpanExporter` without `span_exporter=`. If a user insists on keeping it and only wants sessions: `instrument_tags({"gen_ai.conversation.id": conversation_id})` around `agent.run()` groups traces (dotted tag keys become span attributes verbatim). - Span layout the processor fixes: `BaseWorkflowAgent.run_agent_step` → `._prepare_chat_with_tools` (kind LLM, ~1 ms) + `.astream_chat` (OpenAILike override) → `.astream_chat` (inner, holds messages+usage). Classes implementing the call directly (`OpenAI`) produce two spans. Merge only applies to directly nested identical names, so `CondensePlusContextChatEngine.chat` → `OpenAI.chat` is untouched. Tokens were never doubled (only innermost has usage), only call counts. @@ -231,15 +228,3 @@ More details (from the human guide, for edge cases): - Streaming query engines on llama-index-core 0.14.25: the model call of a `StreamingResponse` runs after the query span ended, so it lands in a separate trace. Fix pending in OpenInference PR #3841 (https://github.com/Arize-ai/openinference/pull/3841); until released, pin `llama-index-core<0.14.25` if the app traces streaming query engines. Agents unaffected. Without Maple access, both must hold: the run exits with no export errors on stderr (`Failed to export`, 401 lines), AND a temporary `SimpleSpanProcessor(ConsoleSpanExporter())` passed into `LlamaIndexForMaple` shows the agent, model and `FunctionTool.acall` spans with `gen_ai.conversation.id` identical on every span of a conversation and `gen_ai.agent.name` set. Silence alone proves nothing (no spans is silent too). With the Maple MCP: `list_agent_sessions` with `search=` returns one row. - -## Do not - -- Do not use or keep `LlamaIndexOpenTelemetry` (`llama-index-observability-otel`) for Maple, and never alongside the OpenInference instrumentor. -- Do not add the exporter to the provider directly; always through `LlamaIndexForMaple`. -- Do not treat `Context` or `llamaindex.run_id` as a session id; use `using_session` around `run()`. -- Do not generate a new conversation id per request, and do not share one `Context` across conversations. -- Do not wrap `yield` statements in `using_session`/`instrument_tags` blocks. -- Do not add provider instrumentors (OpenAI/LiteLLM OpenInference, OpenLLMetry) on top: duplicate model spans and tokens. -- Do not add `maple_ai.session.id` (re-vendors spans) or custom pricing attributes. -- Do not return error strings from failing tools where a raise is acceptable. -- Do not print or commit real keys beyond the repo's convention. diff --git a/skills/maple-agent-tracing-mastra/SKILL.md b/skills/maple-agent-tracing-mastra/SKILL.md index 4c80fe222..77b2f744e 100644 --- a/skills/maple-agent-tracing-mastra/SKILL.md +++ b/skills/maple-agent-tracing-mastra/SKILL.md @@ -7,9 +7,7 @@ description: "Trace Mastra agents and workflows with Maple: export Mastra's buil Goal: every conversation = one Maple Agent Session. Each `agent.generate()` / `agent.stream()` / workflow `run.start()` = one turn (one trace) with transcript, `chat ` spans with tokens, `execute_tool ` spans with args/results, failed tools marked failed, sub-agents in their own lanes. -Human guide with the reasoning: https://maple.dev/docs/agent-tracing/mastra - -Mechanism: Mastra's own tracing (`@mastra/observability`) converted to OTel GenAI semconv v1.38 by `@mastra/otel-exporter`, which runs its own BatchSpanProcessor. No OTel SDK or instrumentation package needed. Maple detects the vendor from resource `telemetry.sdk.name=@mastra/otel-exporter` and reads `gen_ai.conversation.id` as the session key. The exporter writes that key from span `metadata.threadId`, which Mastra sets from `memory.thread`. No thread = no session key. +The session key is `gen_ai.conversation.id`; the exporter writes it from span `metadata.threadId`, which Mastra sets from `memory.thread`. No thread = no session key. Mastra 1.71 has three export gaps that a small span processor (Step 2) fixes; it is required in every setup: the `chat` span has no input messages (empty prompt side of the transcript), sub-agents get their own thread id (turn split, wrong session), and step spans carry the raw provider HTTP response (headers with cookies, full reply body) as `mastra.metadata.headers` / `mastra.metadata.body`, which `hideOutput` does not hide. @@ -29,7 +27,7 @@ Mastra 1.71 has three export gaps that a small span processor (Step 2) fixes; it ## Step 1: Key and region - US: `https://ingest.maple.dev`. EU: `https://ingest.eu.maple.dev`. -- Header: `Authorization: Bearer ` (passed as a headers object in code; no `%20` encoding). +- Header: `Authorization: Bearer ` (passed as a headers object in code). - Key given in the prompt → use it. - No key → use the literal `MAPLE_TEST` (ingest accepts and discards it) and tell the user to replace it with their key from Settings → Ingestion. - Never put a private `maple_sk_` key in browser code. @@ -207,16 +205,5 @@ Without Maple access: `logLevel: "debug"` shows `Export completed` and no `Expor ## Do not -- Do not pass a plain object as `observability`; use `new Observability(...)`. -- Do not rely on `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_EXPORTER_OTLP_HEADERS` / `OTEL_EXPORTER_OTLP_PROTOCOL`; the custom provider ignores them. -- Do not omit `protocol: "http/protobuf"` (defaults to `http/json`, needs a different package). -- Do not add an OTel NodeSDK, `@vercel/otel`, OpenLLMetry or OpenInference just for Mastra; they are unnecessary and double-trace model calls. -- Do not use both OtelExporter and `@mastra/otel-bridge` to reach Maple. -- Do not call `generate()` / `stream()` without a stable `memory.thread` (or `tracingOptions.metadata.threadId`) in a multi-turn chat. -- Do not enable `includeInternalSpans`. -- Do not stamp `maple_ai.session.id` on Mastra spans; it re-vendors them away from Mastra decoding. Use the thread id. -- Do not leave more than one copy of `@mastra/observability` installed (`npm ls @mastra/observability`); the exporter picks the model-call span from features both packages report. -- Do not leave out `mapleSpanProcessor`, and do not replace it with a processor that returns a copy of the span. -- Do not return error objects from tools you want counted as failures; throw. -- Do not exit a script without `await mastra.shutdown()`. -- Do not promise cost or time-to-first-token in Maple: Mastra emits neither under keys Maple reads. +- Do not add an OTel NodeSDK, `@vercel/otel`, OpenLLMetry or OpenInference for Mastra: double-traces model calls. +- Do not stamp `maple_ai.session.id` on Mastra spans: it re-vendors them away from Mastra decoding. diff --git a/skills/maple-agent-tracing-microsoft-agent-framework/SKILL.md b/skills/maple-agent-tracing-microsoft-agent-framework/SKILL.md index 31d209610..983049da7 100644 --- a/skills/maple-agent-tracing-microsoft-agent-framework/SKILL.md +++ b/skills/maple-agent-tracing-microsoft-agent-framework/SKILL.md @@ -5,7 +5,7 @@ description: "Trace Microsoft Agent Framework and Semantic Kernel agents (Python # Maple agent tracing: Microsoft Agent Framework and Semantic Kernel -Goal: one user conversation = one Maple Agent Session, with a transcript, every model call, every tool call (arguments, results, failures) and tokens. Human guide with the reasoning: https://maple.dev/docs/agent-tracing/microsoft-agent-framework +Goal: one user conversation = one Maple Agent Session, with a transcript, every model call, every tool call (arguments, results, failures) and tokens. The framework emits the spans itself. You add: an OTLP/HTTP exporter, content capture, a `gen_ai.conversation.id` span processor (the framework never sets one for local-history sessions), and a flush. @@ -263,12 +263,5 @@ Run one real conversation (2-3 turns, one tool call; a second conversation if ch ## Do not -- Do not leave the MAF protocol at its gRPC default. -- Do not use `gen_ai.agent.id`, `workflow.id`, `service.instance.id` or a module-level constant as the conversation id. - Do not pass `conversation_id` as an agent/run option to label traces: it disables in-memory history and becomes `previous_response_id` on the Responses client. -- Do not call `configure_otel_providers()` when the app already has a TracerProvider, and do not create a second provider. -- Do not set `OTEL_SEMCONV_STABILITY_OPT_IN` without `gen_ai_latest_experimental`. -- Do not add a provider-SDK instrumentor (OpenAI, httpx-based GenAI instrumentors) on top; MAF already traces model calls. -- Do not import `semantic_kernel` before setting its diagnostics env vars. - Do not pass a bare string as `Message("user", text)` contents; use `[text]`. -- Do not filter out non-AI spans (HTTP, workflow) in a collector; dropping parents orphans the AI spans. diff --git a/skills/maple-agent-tracing-openai-agents/SKILL.md b/skills/maple-agent-tracing-openai-agents/SKILL.md index e57ad6ac5..a0243d9ee 100644 --- a/skills/maple-agent-tracing-openai-agents/SKILL.md +++ b/skills/maple-agent-tracing-openai-agents/SKILL.md @@ -7,9 +7,7 @@ description: "Trace OpenAI Agents SDK agents with Maple: bridges the SDK's traci Goal: every conversation with the app shows up in Maple **Agent Sessions** as exactly one session, one turn per `Runner.run`, with transcript, model calls, tool calls (failures marked), agent lanes for sub-agents and handoffs, and tokens (streamed turns included). -Human guide with the reasoning: https://maple.dev/docs/agent-tracing/openai-agents - -Mechanism: the SDK has its own tracing pipeline (not OpenTelemetry) whose default processor uploads to the OpenAI dashboard. `openinference-instrumentation-openai-agents` registers a processor on that pipeline that converts each SDK span into an OTel span; an OTel SDK `TracerProvider` + OTLP/HTTP exporter sends them to Maple. Maple fingerprints the scope `openinference.instrumentation.openai_agents` as "OpenAI Agents SDK" and reads `session.id` for the session. +Mechanism: the SDK has its own tracing pipeline (not OpenTelemetry) whose default processor uploads to the OpenAI dashboard. `openinference-instrumentation-openai-agents` registers a processor on that pipeline that converts each SDK span into an OTel span; an OTel SDK `TracerProvider` + OTLP/HTTP exporter sends them to Maple. Python is the primary path. TypeScript (`@openai/agents` in `package.json`): Step 1 applies; then follow Step 2e in place of Steps 0 and 2a-6, and verify with Step 7. The TypeScript bridge exports less (see the end of 2e). @@ -53,7 +51,7 @@ opentelemetry-exporter-otlp-proto-http>=1.45 OTEL_SERVICE_NAME= OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name= OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev # EU: https://ingest.eu.maple.dev -OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer +OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf ``` @@ -145,7 +143,7 @@ Without it the SDK sends no `stream_options` for non-OpenAI clients and every `r Steps 0, 2a-2d and 3-6 are Python. For TypeScript do this section instead, then Step 7. Find every `run(` / `runner.run(` / `Runner` call and where the conversation id lives first. -Verified 2026-09-29 against a mock OpenAI server and a local OTLP receiver: `@openai/agents` 0.18.0, `@arizeai/openinference-instrumentation-openai-agents` 0.2.15, `@arizeai/openinference-core` 2.7.1, `@opentelemetry/sdk-node` 0.222.0, `@opentelemetry/api` 1.9.1, Node 26. Responses API (default model, streamed and not) and `OpenAIChatCompletionsModel` (streamed and not), two conversations, tool calls, a throwing tool. +Verified 2026-09-29: `@openai/agents` 0.18.0, `@arizeai/openinference-instrumentation-openai-agents` 0.2.15, `@arizeai/openinference-core` 2.7.1, `@opentelemetry/sdk-node` 0.222.0, `@opentelemetry/api` 1.9.1, Node 26. Install (the project's package manager): @@ -200,7 +198,7 @@ Flush: - Script / CLI: `await sdk.shutdown().catch((err) => console.error("telemetry flush failed", err))` in `finally`: `shutdown()` rejects when an export failed, and a Maple outage must not crash the app. - Serverless: add `@opentelemetry/sdk-trace-base` + `@opentelemetry/exporter-trace-otlp-proto`, build `export const spanProcessor = new BatchSpanProcessor(new OTLPTraceExporter())`, pass `new NodeSDK({ spanProcessors: [spanProcessor] })`, and `await spanProcessor.forceFlush()` in `finally` of EVERY invocation (verified: spans arrive with the process exiting right after `forceFlush()`). Never `shutdown()` per invocation. `NodeSDK` has no `forceFlush()` of its own. -What Maple shows for TypeScript (verified against the exported spans and Maple's read path; tell the user): +What Maple shows for TypeScript (tell the user): - Works: one session per conversation id, one turn per `run` (root `Agent workflow` AGENT span, turn label = last user message), operation per span from `openinference.span.kind` (LLM -> chat, TOOL -> execute_tool, AGENT -> invoke_agent), model (`llm.model_name`: the configured id on Chat Completions, the dated snapshot OpenAI returns on Responses), provider `openai`, input/output tokens on every model call INCLUDING streamed Chat Completions on non-OpenAI base URLs (the JS SDK always sends `stream_options.include_usage` when streaming; no `include_usage` step needed), cached tokens on Responses, reasoning tokens on Responses and on Chat Completions (when the provider reports them), tool name, tool failures (a throwing tool's span is status ERROR `Error running tool (non-fatal): ...`; a tool that RETURNS an error string counts as success), prompts in the transcript (from `input.value`). Handoffs are TOOL spans `handoff to ` with tool name `handoff_to_`. - Agent lanes: one per agent name, read from the agent spans' `graph.node.id`. - Missing: cost. @@ -258,7 +256,7 @@ async def handle_message(conversation_id: str, text: str) -> str: - Handoffs: a `handoff to ` span (counted as a tool call named `transfer_to_`, snake-cased, via `MapleSpanFixes`) and the target agent span as a sibling of the source agent's. Nothing to add. - `needs_approval=True` tools: the paused run records a tool span without a result, and the resumed run records the executed call again, so Maple shows the tool twice for one approved call. Expected; tell the user. - Known gaps, don't try to fix: no `gen_ai.tool.call.id` on tool spans; no `gen_ai.response.id` on Chat Completions model spans; no cost. -- Model spans: named `generation` (Chat Completions) or `response` (Responses API). Model on Chat Completions = the configured id (`openai/gpt-4o-mini`); on Responses = the name OpenAI returns, usually a dated snapshot (`gpt-4o-mini-2024-07-18`). Provider is always `openai` (even an Anthropic model behind OpenRouter); cached tokens are inside the input total. Responses API also records cached input tokens and reasoning tokens (`llm.token_count.completion_details.reasoning`). The bridge doesn't export the SDK's per-run/per-turn usage totals, so nothing is double-counted. +- Model spans: named `generation` (Chat Completions) or `response` (Responses API). Model on Chat Completions = the configured id (`openai/gpt-4o-mini`); on Responses = the name OpenAI returns, usually a dated snapshot (`gpt-4o-mini-2024-07-18`). Provider is always `openai` (even an Anthropic model behind OpenRouter); cached tokens are inside the input total. Responses API also records cached input tokens and reasoning tokens (`llm.token_count.completion_details.reasoning`). - Cost via OpenRouter: its Broadcast traces (https://maple.dev/docs/agent-tracing/openrouter) carry per-call cost. Because Chat Completions model spans have no `gen_ai.response.id`, nest the Broadcast spans under them (see "Join Broadcast to your own traces" in that guide) or each call is counted twice. ## Step 6: Flush @@ -299,20 +297,3 @@ Run one real conversation: 2-3 turns with the same conversation id including one Without Maple access, both must hold: the run exits with no export errors on stderr (`Failed to export`, `OTLPExporterError`, 401 lines), AND a local console/in-memory exporter shows the expected span names with `gen_ai.conversation.id` (TS) / `session.id` (Python) on every span. Silence alone proves nothing (no spans is silent too). With the Maple MCP: `list_agent_sessions` with `search=` returns one row. Raw span check (optional, e.g. an `InMemorySpanExporter` in a scratch run): model spans have `gen_ai.operation.name=chat`, `gen_ai.input.messages`, `gen_ai.usage.input_tokens`, `session.id`; tool spans have `gen_ai.operation.name=execute_tool`, `gen_ai.tool.call.arguments` equal to the call's arguments; agent spans have `gen_ai.agent.name`. - -## Do not - -- Do not disable the SDK's tracing (`set_tracing_disabled(True)`, `OPENAI_AGENTS_DISABLE_TRACING`, `tracing_disabled=True`) to stop the OpenAI upload; `set_trace_processors` / the TS bridge already removes it. -- (Python) Do not omit `TraceConfig(enable_genai_semconv=True)`. -- (Python) Do not register `MapleSpanFixes` with `add_trace_processor` or after `instrument()`; it must precede the OpenInference processor. -- (Python) Do not re-add `default_processor()` (or anything via `add_trace_processor`) without a valid OpenAI key: logs fill with `Tracing client error 401`. -- Do not put `tool` in `workflow_name`. -- (Python) Do not pass OpenRouter-style model strings (`Agent(model="openai/gpt-4o-mini")`): the SDK strips `openai/` and rejects other prefixes (`Unknown prefix: anthropic`). Use `OpenAIChatCompletionsModel(model=..., openai_client=...)`. -- (TS) Non-OpenAI base URLs: `setDefaultOpenAIClient(new OpenAI({ baseURL }))` + `setOpenAIAPI("chat_completions")`; model strings like `openai/gpt-5-mini` pass through unchanged. -- Do not rely on `group_id`/`groupId`, `trace_metadata` or SDK `Session` ids for the Maple session; use `using_session` (Python) / `setAttributes` + `context.with` (TS). -- Do not generate a fresh session id per request or use a constant. -- Do not stack `openinference-instrumentation-openai`, Logfire, Langfuse, Traceloop or the OTel contrib Agents instrumentation on the same process. -- (Python) Do not skip `include_usage=True` for streamed non-OpenAI Chat Completions models. -- Do not create a second `TracerProvider` / `NodeSDK`, or call `instrument()` twice. -- Do not use `SimpleSpanProcessor` or a console exporter in servers. -- Do not skip the flush in scripts, notebooks, CLIs and serverless. diff --git a/skills/maple-agent-tracing-openrouter/SKILL.md b/skills/maple-agent-tracing-openrouter/SKILL.md index 2f257fcb9..822cc04a9 100644 --- a/skills/maple-agent-tracing-openrouter/SKILL.md +++ b/skills/maple-agent-tracing-openrouter/SKILL.md @@ -7,9 +7,7 @@ description: "Trace OpenRouter calls with Maple: route OpenRouter Broadcast (OTL Goal: every conversation = one Maple Agent Session with each OpenRouter model call, its tokens, cost (`gen_ai.usage.total_cost`, OpenRouter's real charge) and prompt/completion. If the app already exports its own traces to Maple, the Broadcast spans nest inside them and each call is counted once. -Human guide with the reasoning: https://maple.dev/docs/agent-tracing/openrouter - -Mechanism: OpenRouter Broadcast, configured in the OpenRouter dashboard, exports one OTLP/HTTP JSON trace per request (scope and `service.name` = `openrouter`, root span `LLM Generation`, children `provider attempt N: `). Maple detects vendor `openrouter` from the scope and reads **`session.id`** as the session key. OpenRouter sets `session.id` from the request's `session_id` body field or `x-session-id` header, and uses `trace.trace_id` / `trace.parent_span_id` from the body verbatim as the OTLP trace id / parent span id. +Mechanism: OpenRouter Broadcast, configured in the OpenRouter dashboard, exports one OTLP/HTTP JSON trace per request (scope and `service.name` = `openrouter`, root span `LLM Generation`, children `provider attempt N: `). Maple reads **`session.id`** as the session key. OpenRouter sets `session.id` from the request's `session_id` body field or `x-session-id` header, and uses `trace.trace_id` / `trace.parent_span_id` from the body verbatim as the OTLP trace id / parent span id. You cannot change the OpenRouter dashboard. Your job: (1) edit the app's OpenRouter calls, (2) hand the user the exact destination settings. @@ -169,16 +167,4 @@ Tell the user: cost is shown (OpenRouter's charge); TTFT, environment, tool call - `service.name` on Broadcast spans is always `openrouter`, no environment attribute. Custom keys in the `trace` object arrive as `trace.metadata.` (searchable in Traces, don't set service/environment). - OpenRouter's sample trace (`Test Trace - OpenRouter Observability`, an `openai/gpt-4-turbo` call with sample tokens/cost) can appear as a session; ignore it. - Test Connection passes but nothing arrives: check API key filter, data regions, **Enable Broadcast** on the account/org the app key belongs to, and that the key isn't `MAPLE_TEST`. -- Destinations can be created via OpenRouter's observability API (`type: "otel-collector"`) with a management key (only if the user asks; see Do not). - -## Do not - -- Do not use a constant or per-client `session_id`; it merges every user into one session. -- Do not use non-hex or wrong-length `trace_id` / `parent_span_id`; use the active span's W3C ids. -- Do not use a different id for `session_id` than the framework's conversation id when both exist. -- Do not stack Broadcast on in-app instrumentation without nesting or a response-id match; tokens and calls double. -- Do not claim Broadcast captures tools, tool errors or sub-agents. -- Do not point the destination at `https://ingest.maple.dev` without `/v1/traces`, and do not put the Maple key anywhere in the repo for Broadcast. -- Do not set sampling below 1 to "save volume" without telling the user whole sessions disappear. -- Do not create or edit OpenRouter destinations via the management API unless the user explicitly gives a management key and asks. -- Do not put PII in `user`, `session_id` or `trace` metadata. +- Destinations can be created via OpenRouter's observability API (`type: "otel-collector"`) with a management key (only if the user explicitly asks and provides a management key). diff --git a/skills/maple-agent-tracing-opentelemetry/SKILL.md b/skills/maple-agent-tracing-opentelemetry/SKILL.md index 1df1aafcb..1377df209 100644 --- a/skills/maple-agent-tracing-opentelemetry/SKILL.md +++ b/skills/maple-agent-tracing-opentelemetry/SKILL.md @@ -7,14 +7,12 @@ description: "Trace a hand-rolled or unsupported AI agent with Maple by emitting Goal: every conversation = one Maple Agent Session. Each user message = one trace rooted at an `invoke_agent` span, with a `chat` span per model call (model, tokens, transcript) and an `execute_tool` span per tool call (args, result, failures). -Human guide with the reasoning: https://maple.dev/docs/agent-tracing/opentelemetry - Mechanism: you write the spans. Maple classifies a span only by `gen_ai.operation.name`, groups a trace by `gen_ai.conversation.id`, and reads content only from span attributes. Hand-written spans show as framework "Unidentified" (vendor `unknown:genai`); that is expected. ## Step 0: Detect 1. Language and entry points (web server, workers, scripts, serverless handlers). -2. Is a supported framework the real agent runtime? (`@mastra/core`, `ai`, `@openai/agents`/`openai-agents`, `langchain`/`langgraph`, `pydantic-ai`, `crewai`, `google-adk`, `llama-index`, `strands-agents`, `smolagents`, `agno`, `dspy`, `haystack-ai`, `agent-framework`, Spring AI, `litellm`, Claude Agent SDK). If yes, stop and use `maple-agent-tracing-` instead; use this skill only for the parts that framework doesn't cover, or for the `maple_ai.session.id` wrapper (Step 4). +2. Is a supported framework the real agent runtime? (`@mastra/core`, `ai`, `agents`/`@cloudflare/ai-chat`, `genkit`/`@genkit-ai/*`, `@openai/agents`/`openai-agents`, `langchain`/`langgraph`, `pydantic-ai`, `crewai`, `google-adk`, `llama-index`, `strands-agents`, `smolagents`, `agno`, `dspy`, `haystack-ai`, `agent-framework`, Spring AI, `litellm`, Claude Agent SDK). If yes, stop and use `maple-agent-tracing-` instead; use this skill only for the parts that framework doesn't cover, or for the `maple_ai.session.id` wrapper (Step 4). 3. Existing OTel setup. Search for `TracerProvider`, `NodeTracerProvider`, `NodeSDK`, `registerOTel`, `set_tracer_provider`, `opentelemetry-instrument`, `logfire.configure`, `sentry_sdk.init`/`Sentry.init`, `otel.SetTracerProvider`. Exists → add Maple's exporter/processor to it; never create a second provider. 4. Existing GenAI auto-instrumentation on the model client (`@opentelemetry/instrumentation-openai`, `opentelemetry-instrumentation-openai-v2`, OpenLLMetry `Traceloop.init`, OpenInference `OpenAIInstrumentor`, `logfire.instrument_openai`). Pick one source of `chat` spans: either keep that instrumentation (then see `maple-agent-tracing-provider-sdks`) or remove it and write `chat` spans here. Both = every model call twice. 5. Find in the code: the agent loop (where one user message is handled), every model call site, every tool dispatch, sub-agent calls, and where the conversation/chat/thread id lives in the request. @@ -30,11 +28,11 @@ Mechanism: you write the spans. Maple classifies a span only by `gen_ai.operatio ```bash OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev -OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer +OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf ``` -The exporters append `/v1/traces`. If an SDK rejects the space in the header, use `Bearer%20`. +The exporters append `/v1/traces`. - The SDK exporters read these env vars when the exporter is constructed. If the app loads `.env` (dotenv, `load_dotenv()`, `--env-file`), load it at the top of the tracing module, before the provider is built; otherwise the exporter silently targets `localhost:4318` with no key. - If the header is built from your own env var, never let an unset var become `Bearer undefined` (opaque 401) or a bare `KeyError` on import: fail fast with a clear message, or inline the key when the repo has no env convention. @@ -45,7 +43,7 @@ The exporters append `/v1/traces`. If an SDK rejects the space in the header, us Read the reference for the language and adapt it: - TypeScript/Node: `references/typescript.md` - Python: `references/python.md` -- Go, Rust, Ruby, Elixir, Java, .NET: `references/go.md` (Go code + notes for the others) +- Other languages: the language's OTel SDK with an OTLP/HTTP exporter, following the steps below. Rules: - Init module is imported first in every entry point. Set a real `service.name` and `deployment.environment.name`. @@ -69,7 +67,7 @@ Rules: Message JSON (`input.messages`/`output.messages`): array of `{role, parts}`; parts `{type:"text",content}`, `{type:"tool_call",id,name,arguments:}`, `{type:"tool_call_response",id,response}`, `{type:"reasoning",content}`. Output messages add `finish_reason`. `system_instructions` = array of parts, no role: `[{"type":"text","content":"..."}]`. Always a JSON **string** attribute; plain-text messages don't render. -Also read: `reasoning` parts are rendered; a message may carry `content` (string or part array) instead of `parts`. `gen_ai.response.model` wins over `gen_ai.request.model` when both are set. `gen_ai.response.finish_reasons` feeds the refusal (`content_filter`) and truncation (`length`) checks. +Also read: a message may carry `content` (string or part array) instead of `parts`. `gen_ai.response.model` wins over `gen_ai.request.model` when both are set. Legacy spellings are read as fallbacks (use current names in new code): `gen_ai.system` (→ `gen_ai.provider.name`, renamed in semconv 1.37), `gen_ai.usage.prompt_tokens`/`completion_tokens`, whole-value `gen_ai.prompt`/`gen_ai.completion`, `gen_ai.usage.cache_creation.input_tokens` (→ `cache_write`), `gen_ai.usage.total_cost` (→ `cost`). @@ -120,7 +118,7 @@ await tracer.startActiveSpan( ## Step 6: Tools, errors, sub-agents -- Tool failure: set status ERROR with the error message as description, set `error.type` (exception class or error code), no `gen_ai.tool.call.result`, then return the error to the model as the tool result so the loop continues. Keep the message specific: Maple's tool pages group failures by it (ids and numbers masked). +- Tool failure: set status ERROR with the error message as description, set `error.type` (exception class or error code), no `gen_ai.tool.call.result`, then return the error to the model as the tool result so the loop continues. Keep the message specific: Maple's tool pages group failures by it. - Maple counts any span as failed if it has status ERROR, a non-empty `error.type`, or `gen_ai.response.status`=`failed`. - Tools that return `{"error": ...}` instead of raising: mark the span failed the same way when you detect it. - Model call failure: status ERROR + `error.type` (HTTP status or exception class), rethrow. @@ -143,7 +141,6 @@ await tracer.startActiveSpan( - Node script/CLI: `await provider.shutdown()` in `finally`. Serverless: `await provider.forceFlush()` before returning (inside `waitUntil`/`after()` if available). Both reject when an export failed: add `.catch((err) => console.error("telemetry flush failed", err))` so a Maple outage can't crash the app. Long-running server: flush on `SIGTERM`, nothing per request. - Python script: `provider.shutdown()` in `finally`. Lambda: `force_flush()` in `finally`. Notebooks/workers: `force_flush()` per cell/task. -- Go: `defer tp.Shutdown(context.Background())` in `main`. ## Step 9: Verify @@ -164,19 +161,3 @@ Run one real conversation: 2+ messages with the same id, one streamed reply if t With the Maple MCP: `list_agent_sessions` with `search=` returns one row. Check without Maple access: the run exits with no export errors on stderr (`Failed to export`, `OTLPExporterError`, 401 lines) AND a temporary console exporter (`ConsoleSpanExporter` + `SimpleSpanProcessor`) shows the expected span tree with `gen_ai.conversation.id`, and every messages attribute parses with `JSON.parse`/`json.loads`. Silence alone proves nothing: no spans also looks silent. - -## Do not - -- Do not emit spans without `gen_ai.operation.name` and expect them in Agent Sessions. -- Do not send messages as plain text, structured attribute values, span events or logs. -- Do not generate a conversation id per request or use the trace id. -- Do not give sub-agents their own conversation ids. -- Do not set attribute length limits. -- Do not stack a provider auto-instrumentor on top of hand-written `chat` spans. -- Do not put `maple_ai.session.id` on framework spans or on `chat` spans. -- Do not send Anthropic's raw `input_tokens` or Gemini's raw `candidatesTokenCount` as the totals (Step 7). -- Do not put usage on `invoke_agent` spans. -- Do not use `gen_ai.system` in new code (read as a fallback only); use `gen_ai.provider.name`. -- Do not name the tracer after a framework or gateway. -- Do not create a second TracerProvider. -- Do not print or commit real keys beyond the repo's convention. diff --git a/skills/maple-agent-tracing-opentelemetry/references/go.md b/skills/maple-agent-tracing-opentelemetry/references/go.md deleted file mode 100644 index 00e42fabc..000000000 --- a/skills/maple-agent-tracing-opentelemetry/references/go.md +++ /dev/null @@ -1,155 +0,0 @@ -# Go reference - -Pattern for `go.opentelemetry.io/otel` v1.46 (`sdk`, `exporters/otlp/otlptrace/otlptracehttp` v1.46). Not compiled in authoring; run `go vet ./...` after adding it. - -```bash -go get go.opentelemetry.io/otel go.opentelemetry.io/otel/sdk go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp -``` - -Wrap the project's existing client call in `Chat` and tool dispatch in `Tool`. Pass the `ctx` returned by `StartTurn` into both, or the spans become separate traces. Set `gen_ai.provider.name` to the API actually called. Copy cache/reasoning counts when the provider returns them (`gen_ai.usage.cache_read.input_tokens`, `gen_ai.usage.reasoning.output_tokens`). - -```go -// genai.go -package agent - -import ( - "context" - "encoding/json" - "fmt" - - "go.opentelemetry.io/otel" - "go.opentelemetry.io/otel/attribute" - "go.opentelemetry.io/otel/codes" - "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" - "go.opentelemetry.io/otel/sdk/resource" - sdktrace "go.opentelemetry.io/otel/sdk/trace" - "go.opentelemetry.io/otel/trace" -) - -var tracer = otel.Tracer("support-agent") - -// SetupTracing reads OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS. -// Call Shutdown on the returned provider before the process exits. -func SetupTracing(ctx context.Context) (*sdktrace.TracerProvider, error) { - exporter, err := otlptracehttp.New(ctx) - if err != nil { - return nil, err - } - tp := sdktrace.NewTracerProvider( - sdktrace.WithBatcher(exporter), - sdktrace.WithResource(resource.NewSchemaless( - attribute.String("service.name", "support-agent"), - attribute.String("deployment.environment.name", "production"), - )), - ) - otel.SetTracerProvider(tp) - return tp, nil -} - -// Message is the GenAI semconv shape: {role, parts}. -type Message struct { - Role string `json:"role"` - Parts []map[string]any `json:"parts"` - FinishReason string `json:"finish_reason,omitempty"` -} - -// ChatResult holds what your provider returned, copied verbatim. -type ChatResult struct { - ID, Model, FinishReason string - Output Message - InputTokens, OutputTokens int64 - CostUSD float64 // 0 when the provider doesn't return a cost -} - -func jsonAttr(key string, value any) attribute.KeyValue { - b, _ := json.Marshal(value) - return attribute.String(key, string(b)) -} - -func fail(span trace.Span, err error) { - span.SetStatus(codes.Error, err.Error()) - span.SetAttributes(attribute.String("error.type", fmt.Sprintf("%T", err))) -} - -// StartTurn opens the invoke_agent span for one user message. End it when the turn is done. -func StartTurn(ctx context.Context, agentName, conversationID string) (context.Context, trace.Span) { - return tracer.Start(ctx, "invoke_agent "+agentName, trace.WithAttributes( - attribute.String("gen_ai.operation.name", "invoke_agent"), - attribute.String("gen_ai.agent.name", agentName), - attribute.String("gen_ai.conversation.id", conversationID), - )) -} - -// Chat wraps one model call (your existing client code goes in call). -func Chat(ctx context.Context, model string, input []Message, call func(context.Context) (ChatResult, error)) (ChatResult, error) { - ctx, span := tracer.Start(ctx, "chat "+model, trace.WithSpanKind(trace.SpanKindClient), trace.WithAttributes( - attribute.String("gen_ai.operation.name", "chat"), - attribute.String("gen_ai.provider.name", "openai"), - attribute.String("gen_ai.request.model", model), - jsonAttr("gen_ai.input.messages", input), - )) - defer span.End() - res, err := call(ctx) - if err != nil { - fail(span, err) - return res, err - } - res.Output.FinishReason = res.FinishReason - span.SetAttributes( - attribute.String("gen_ai.response.id", res.ID), - attribute.String("gen_ai.response.model", res.Model), - attribute.StringSlice("gen_ai.response.finish_reasons", []string{res.FinishReason}), - attribute.Int64("gen_ai.usage.input_tokens", res.InputTokens), - attribute.Int64("gen_ai.usage.output_tokens", res.OutputTokens), - jsonAttr("gen_ai.output.messages", []Message{res.Output}), - ) - if res.CostUSD > 0 { - span.SetAttributes(attribute.Float64("gen_ai.usage.cost", res.CostUSD)) - } - return res, nil -} - -// Tool wraps one tool call. result is stored as JSON. -func Tool(ctx context.Context, name, callID, arguments string, run func(context.Context) (any, error)) (string, error) { - ctx, span := tracer.Start(ctx, "execute_tool "+name, trace.WithAttributes( - attribute.String("gen_ai.operation.name", "execute_tool"), - attribute.String("gen_ai.tool.name", name), - attribute.String("gen_ai.tool.call.id", callID), - attribute.String("gen_ai.tool.call.arguments", arguments), - )) - defer span.End() - result, err := run(ctx) - if err != nil { - fail(span, err) - return "", err - } - b, _ := json.Marshal(result) - span.SetAttributes(attribute.String("gen_ai.tool.call.result", string(b))) - return string(b), nil -} -``` - -Usage: - -```go -tp, err := agent.SetupTracing(ctx) -if err != nil { - log.Fatal(err) -} -defer tp.Shutdown(context.Background()) - -ctx, turn := agent.StartTurn(ctx, "support", chatID) -defer turn.End() -res, err := agent.Chat(ctx, model, input, func(ctx context.Context) (agent.ChatResult, error) { - return callModel(ctx, model, input) // your existing client code -}) -``` - -Failure on the turn span: call the same `fail(turn, err)` pattern before returning an error. - -`otlptracehttp.New` reads `OTEL_*` when it runs. If the app loads `.env` (`godotenv.Load()`), do it before `SetupTracing`; otherwise the exporter silently targets `localhost:4318` with no key. No env convention: pass `otlptracehttp.WithEndpointURL("https://ingest.maple.dev/v1/traces")` and `otlptracehttp.WithHeaders(map[string]string{"Authorization": "Bearer "})` inline. - -## Other languages (Rust, Ruby, Elixir, Java, .NET) - -Same three spans, same attribute keys and value types (string, int64, double, string array). Serialize every messages/tool payload to a JSON string before setting it. Register the SDK's context propagation so child spans nest (Rust: `Context::current_with_span` / `tracing-opentelemetry`; Ruby: `in_span`; Elixir: `OpenTelemetry.Tracer.with_span`). - diff --git a/skills/maple-agent-tracing-provider-sdks/SKILL.md b/skills/maple-agent-tracing-provider-sdks/SKILL.md index c6651ba71..dcb26d8c8 100644 --- a/skills/maple-agent-tracing-provider-sdks/SKILL.md +++ b/skills/maple-agent-tracing-provider-sdks/SKILL.md @@ -7,13 +7,10 @@ description: "Trace agents built directly on the OpenAI, Anthropic or Google Gen Goal: every conversation = one Maple Agent Session. Each user message = one turn = one trace rooted at an `invoke_agent ` span, containing a `chat ` / `generate_content ` span per model call (transcript + tokens) and an `execute_tool ` span per tool call (args, result, failures). -Human guide with the reasoning: https://maple.dev/docs/agent-tracing/provider-sdks - Mechanism: - Python: OpenTelemetry GenAI instrumentations (`opentelemetry-instrumentation-genai-openai`, `-genai-anthropic`, `opentelemetry-instrumentation-google-genai`, all >= 1.2b0) write `chat` spans with GenAI semconv on span attributes. - TypeScript: no usable instrumentation (`@opentelemetry/instrumentation-openai` only patches `openai` < 7 and puts messages in log events; nothing official for `@anthropic-ai/sdk` / `@google/genai`). Record the model call with the helper in `references/typescript.md`. -- Both: YOU add the `invoke_agent` span (with `gen_ai.conversation.id`) and `execute_tool` spans. Instrumentations can't see turns, conversations or your tools. -- Maple files these spans under vendor `unknown:genai` (UI: "Unidentified") and reads `gen_ai.conversation.id` as the session key. Everything else is read in full. +- Both: YOU add the `invoke_agent` span (with `gen_ai.conversation.id`) and `execute_tool` spans. ## Step 0: Detect @@ -39,7 +36,7 @@ Env vars (the exporter reads them and appends `/v1/traces`): ```bash OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev -OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer +OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY ``` @@ -141,13 +138,4 @@ Check without Maple access: the run exits with no export errors on stderr (`Fail ## Do not -- Do not install `opentelemetry-instrumentation-openai` or `opentelemetry-instrumentation-anthropic` (OpenLLMetry) or the deprecated `opentelemetry-instrumentation-openai-v2`; install the `-genai-` packages. -- Do not use `@opentelemetry/instrumentation-openai` in TS (no `openai` 7 support; content only in logs). -- Do not run two instrumentations on the same SDK, and do not add provider instrumentation under an agent framework. -- Do not create a second `TracerProvider`. -- Do not set content capture to `EVENT_ONLY` or `true`, or rely on log export for content. -- Do not generate a conversation id per request or share one across conversations. -- Do not give a sub-agent the orchestrator's agent name, or a different conversation id. -- Do not catch tool exceptions outside `run_tool` without marking the span failed. -- Do not stream OpenAI Chat Completions without `stream_options.include_usage`. -- Do not print or commit real keys beyond the repo's convention. +- Do not install `opentelemetry-instrumentation-openai` / `-anthropic` (OpenLLMetry) or the deprecated `-openai-v2`; install the `-genai-` packages. diff --git a/skills/maple-agent-tracing-provider-sdks/references/python.md b/skills/maple-agent-tracing-provider-sdks/references/python.md index 3d47a22f5..b0c30dc3e 100644 --- a/skills/maple-agent-tracing-provider-sdks/references/python.md +++ b/skills/maple-agent-tracing-provider-sdks/references/python.md @@ -53,7 +53,6 @@ from opentelemetry.trace import Status, StatusCode tracer = trace.get_tracer("support-agent") -# Same switch the instrumentors read, so one env var controls content everywhere. CAPTURE_CONTENT = os.environ.get( "OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT", "" ).upper() in ("SPAN_ONLY", "SPAN_AND_EVENT") diff --git a/skills/maple-agent-tracing-provider-sdks/references/typescript.md b/skills/maple-agent-tracing-provider-sdks/references/typescript.md index 656b11998..59469ec0f 100644 --- a/skills/maple-agent-tracing-provider-sdks/references/typescript.md +++ b/skills/maple-agent-tracing-provider-sdks/references/typescript.md @@ -37,7 +37,6 @@ import type OpenAI from "openai" const tracer = trace.getTracer("support-agent") -// Same switch as the Python instrumentors, so one env var controls content everywhere. const captureContent = ["SPAN_ONLY", "SPAN_AND_EVENT"].includes( (process.env.OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT ?? "").toUpperCase(), ) @@ -235,7 +234,6 @@ export function chatTurn( } ``` -- `onText` switches `tracedChat` to streaming with `stream_options.include_usage` and records time to first chunk. - Replace every `client.chat.completions.create(...)` in the agent with `tracedChat(client, params)`. Don't wrap `create` twice. - Sub-agent inside a tool: `runTool(id, "ask_weather_worker", args, (a) => agentSpan("weather_worker", undefined, () => workerLoop(a)))`. diff --git a/skills/maple-agent-tracing-pydantic-ai/SKILL.md b/skills/maple-agent-tracing-pydantic-ai/SKILL.md index f5e1ca19d..ff27a2001 100644 --- a/skills/maple-agent-tracing-pydantic-ai/SKILL.md +++ b/skills/maple-agent-tracing-pydantic-ai/SKILL.md @@ -7,8 +7,6 @@ description: "Trace Pydantic AI agents with Maple: export Pydantic AI's built-in Goal: every conversation = one Maple Agent Session. Each `agent.run()` = one turn (one trace) with transcript, `chat` spans with tokens, `execute_tool` spans with args/results, failed tools marked failed, sub-agents in their own lanes. -Human guide with the reasoning: https://maple.dev/docs/agent-tracing/pydantic-ai - Mechanism: Pydantic AI's native OTel instrumentation (scope `pydantic-ai`, GenAI semconv on span attributes). No extra instrumentation package. Maple reads `gen_ai.conversation.id` as the session key for this framework. ## Step 0: Detect @@ -38,7 +36,7 @@ Env vars (the exporter reads them when it is built; it appends `/v1/traces`). If ```bash OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev -OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer +OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf ``` @@ -211,12 +209,5 @@ Without Maple access (or with `MAPLE_TEST`), both must hold; silence alone prove ## Do not -- Do not rely on the automatic `gen_ai.conversation.id`: it's a new UUID7 per run without history. -- Do not forget `conversation_id=ctx.conversation_id` on delegated runs. -- Do not create a second `TracerProvider` when one exists, and do not add Logfire just for Maple. -- Do not set `version=2|3|4` (deprecated) or `event_mode="logs"` (content moves to logs, which Maple doesn't read). -- Do not add `session.id` or `maple_ai.session.id` attributes: Maple reads `gen_ai.conversation.id` for Pydantic AI, and `maple_ai.session.id` would re-vendor the span. -- Do not also instrument the model client (OpenAI/Anthropic instrumentors, `logfire.instrument_openai`, a global `logfire.instrument_httpx()`): duplicate model-call spans. `instrument_httpx(client)` on a tool's own client is fine. -- Do not return error strings from tools that failed; raise `ToolFailed`. -- Do not add token pricing code. -- Do not print or commit real keys beyond the repo's convention. +- Do not keep `event_mode="logs"`: content moves to logs, which Maple doesn't read (empty transcript). +- Do not add `session.id` or `maple_ai.session.id`: Maple reads `gen_ai.conversation.id` here; `maple_ai.session.id` re-vendors the span. diff --git a/skills/maple-agent-tracing-smolagents/SKILL.md b/skills/maple-agent-tracing-smolagents/SKILL.md index e8745e3c2..c989929ec 100644 --- a/skills/maple-agent-tracing-smolagents/SKILL.md +++ b/skills/maple-agent-tracing-smolagents/SKILL.md @@ -5,9 +5,9 @@ description: "Trace Hugging Face smolagents agents with Maple: OpenInference ins # Maple agent tracing: smolagents -Goal: every conversation the app runs through a smolagents agent shows up in Maple **Agent Sessions** as ONE session, with the transcript, each model call (model, tokens), each tool call (name, arguments, result, failure) and one lane per managed agent. Reasoning and background for every step: https://maple.dev/docs/agent-tracing/smolagents +Goal: every conversation the app runs through a smolagents agent shows up in Maple **Agent Sessions** as ONE session, with the transcript, each model call (model, tokens), each tool call (name, arguments, result, failure) and one lane per managed agent. -smolagents has no OpenTelemetry code of its own. All spans come from `openinference-instrumentation-smolagents`. Its defaults are wrong for Maple in two ways this skill fixes: no session id, and no agent names / wrong tool arguments. +All spans come from `openinference-instrumentation-smolagents`. ## Step 0: Detect versions and existing OpenTelemetry @@ -62,8 +62,6 @@ from opentelemetry.sdk.trace.export import BatchSpanProcessor class SmolagentsForMaple(SpanProcessor): - """Fixes what the smolagents instrumentor gets wrong for Maple: agent names, tool names and arguments.""" - def on_start(self, span, parent_context=None): if span.instrumentation_scope.name != "openinference.instrumentation.smolagents": return @@ -93,7 +91,7 @@ SmolagentsInstrumentor().instrument( - `import tracing` at the top of every entry point (web app module, worker, CLI main) so `instrument()` runs before the first `agent.run()`. Import order relative to `smolagents` does not matter. - Existing provider: skip the `TracerProvider()`/`set_tracer_provider` lines, add `SmolagentsForMaple()` and the exporter to the existing provider, pass it as `tracer_provider=`. -- `enable_genai_semconv=True`: recommended (emits standard GenAI attributes). The env var `OPENINFERENCE_ENABLE_GENAI_SEMCONV=true` is equivalent only if set before `TraceConfig` is constructed; prefer the code form. +- `enable_genai_semconv=True`: required (emits the standard GenAI attributes the processor relies on). The env var `OPENINFERENCE_ENABLE_GENAI_SEMCONV=true` is equivalent only if set before `TraceConfig` is constructed; prefer the code form. - Keep `SmolagentsForMaple` exactly: it must run in `on_start` (before the dual-write, which never overwrites existing keys). ## Step 3: One session per conversation @@ -180,11 +178,4 @@ If sessions are split per message: `using_session` missing or id changing. Nothi ## Do not -- Do not use the `smolagents[telemetry]` extra or `phoenix.otel.register()` to send to Maple. -- Do not add `openinference-instrumentation-openai` / `-litellm` alongside (duplicate model spans, doubled tokens). -- Do not generate a session id per request or use a constant one. -- Do not pass `OTLPSpanExporter(endpoint="https://ingest.maple.dev")` without `/v1/traces`. -- Do not create a second `TracerProvider` when one exists, and do not call `instrument()` twice. -- Do not override `generate` in a custom `Model` subclass and expect model spans. -- Do not rely on `hide_inputs` alone for PII (`smolagents.task`). -- Do not wrap tools in try/except that returns error strings; let them raise. +- Do not use `phoenix.otel.register()` (the smolagents docs' setup) to send to Maple: it skips `SmolagentsForMaple` (no lanes, tools named `SimpleTool`). diff --git a/skills/maple-agent-tracing-spring-ai/SKILL.md b/skills/maple-agent-tracing-spring-ai/SKILL.md index 48f8a0dc7..7378ac38e 100644 --- a/skills/maple-agent-tracing-spring-ai/SKILL.md +++ b/skills/maple-agent-tracing-spring-ai/SKILL.md @@ -7,9 +7,7 @@ description: "Trace Spring AI agents with Maple: wires Spring Boot's OpenTelemet Goal: every conversation with the Spring AI app shows up in Maple **Agent Sessions** as exactly one session, one turn per `ChatClient` call, with transcript, model calls, tool calls (failures marked), sub-agent lanes and tokens. -Human guide with the reasoning: https://maple.dev/docs/agent-tracing/spring-ai - -Mechanism: Spring AI's Micrometer Observations → `micrometer-tracing-bridge-otel` → OpenTelemetry SDK → OTLP/HTTP to Maple, all from `spring-boot-starter-opentelemetry`. Maple detects the spans as Spring AI by their `spring.ai.*` keys. Out of the box: sampling is 10%, prompts/replies never reach spans (`log-prompt`/`log-completion` only log to SLF4J), and thrown tool errors end the span OK. Steps 2-5 fix all three. +Mechanism: Spring AI's Micrometer Observations → `micrometer-tracing-bridge-otel` → OpenTelemetry SDK → OTLP/HTTP to Maple, all from `spring-boot-starter-opentelemetry`. Out of the box: sampling is 10%, prompts/replies never reach spans (`log-prompt`/`log-completion` only log to SLF4J), and thrown tool errors end the span OK. Steps 2-5 fix all three. ## Step 0: Detect versions and existing setup @@ -25,7 +23,7 @@ Mechanism: Spring AI's Micrometer Observations → `micrometer-tracing-bridge-ot - US: `https://ingest.maple.dev`. EU: `https://ingest.eu.maple.dev`. - Header: `Authorization=Bearer `. Protocol: OTLP/HTTP protobuf (Boot's default transport). - Key in the user's prompt: use it. No key: use the literal `MAPLE_TEST` (ingest accepts and discards it) and tell the user to replace it with their key from **Settings → Ingestion**. -- Private `maple_sk_` keys never go in browser code. Spring AI runs server-side; an ingest key is write-only. +- Private `maple_sk_` keys never go in browser code. - Follow the repo's existing secret/env convention (`${ENV_VAR}` placeholders, profile files, Vault/Config Server). If there is none, inline in `application.properties` is acceptable because ingest keys are write-only. - Keep `${MAPLE_INGEST_KEY}` without a default: unresolved, Boot fails at startup with a clear error, while `${MAPLE_INGEST_KEY:}` sends an empty `Bearer ` and gets an opaque 401. Boot does not read `.env` files: export the variable, or add `spring.config.import=optional:file:.env[.properties]` if the repo keeps one. @@ -198,7 +196,7 @@ public class MapleAiObservationConfig { } ``` -Boot applies `ObservationFilter` beans to the registry automatically; nothing else to register. The filter runs when each observation stops, after Spring AI's own conventions. `spring.ai.tools.observations.include-content` writes `spring.ai.tool.call.arguments/result`, which Maple does not read; the filter's `gen_ai.tool.call.*` keys are the ones read. +Boot applies `ObservationFilter` beans to the registry automatically; nothing else to register. `spring.ai.tools.observations.include-content` writes `spring.ai.tool.call.arguments/result`, which Maple does not read; the filter's `gen_ai.tool.call.*` keys are the ones read. Content notes: every `chat` span carries the whole conversation so far, so spans grow with long chats (don't cap them; see 2b). With `maple.ai.capture-content=false` no message/tool content leaves the process; to redact instead, mask values inside `message(...)`. The conversation id and agent names are sent regardless: keep personal data out of them. @@ -257,7 +255,6 @@ public class Workers { ## Step 5b: Tokens and cost (no action needed, explain if asked) -- `chat` spans carry `gen_ai.usage.input_tokens`, `output_tokens`, and `cache_read.input_tokens` / `cache_creation.input_tokens` when the provider reports them; Maple reads all four. `chat_client` spans carry no usage, so nothing is double counted. - Spring AI records the provider as `gen_ai.system`, derived from the client class, not the model: a Claude model behind OpenRouter via the OpenAI starter is labeled `openai`. - No cost attribute is emitted; sessions show as unpriced (Maple never prices tokens). @@ -291,12 +288,4 @@ Without Maple access: the run logs no OTLP export errors or 401s AND a temporary ## Do not -- Do not leave `management.tracing.sampling.probability` at the default `0.1`. -- Do not rely on `log-prompt`/`log-completion`/`include-content` for the transcript; Maple reads `gen_ai.input.messages`/`gen_ai.output.messages`/`gen_ai.tool.call.*` span attributes only. - Do not stamp `maple_ai.session.id` on Spring AI spans; the conversation id param is the supported path. -- Do not put a constant or per-request conversation id on calls. -- Do not register a second `ToolExecutionExceptionProcessor` next to an existing one (ambiguous bean). -- Do not attach the OTel Java agent next to the starter without the `GlobalOpenTelemetry` bean from Step 2c (every span becomes its own trace). -- Do not set an attribute length limit (breaks content JSON). -- Do not use an `ObservationPredicate` to drop Spring AI observations (breaks the streamed turn's trace). -- Do not use this skill for LangChain4j; use the generic OpenTelemetry GenAI guide: https://maple.dev/docs/agent-tracing/opentelemetry diff --git a/skills/maple-agent-tracing-strands/SKILL.md b/skills/maple-agent-tracing-strands/SKILL.md index 019347d54..25231f15a 100644 --- a/skills/maple-agent-tracing-strands/SKILL.md +++ b/skills/maple-agent-tracing-strands/SKILL.md @@ -7,8 +7,6 @@ description: "Trace Strands Agents (AWS, Python or TypeScript) with Maple: expor Goal: every conversation = one Maple Agent Session. Each `agent(...)` / `invoke_async` / `stream_async` call = one turn (one trace) with transcript, `chat` spans with tokens, `execute_tool` spans with args/results, failed tools marked failed, sub-agents in their own lanes. -Human guide with the reasoning: https://maple.dev/docs/agent-tracing/strands - Mechanism: Strands' native OTel tracer (scope `strands.telemetry.tracer`, `gen_ai.provider.name=strands-agents`). No extra instrumentation package. Maple reads `session.id` (then `gen_ai.conversation.id`) as the session key, and reads span ATTRIBUTES only (never span events). ## Step 0: Detect @@ -45,7 +43,7 @@ Env vars. Put them where the repo keeps env (shell/.env/container). `OTEL_SEMCON ```bash OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev -OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer +OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf OTEL_SERVICE_NAME= OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name= @@ -73,7 +71,7 @@ TypeScript. OTel packages are optional peers; install them: npm install @strands-agents/sdk @opentelemetry/api @opentelemetry/sdk-trace-base @opentelemetry/sdk-trace-node @opentelemetry/resources @opentelemetry/exporter-trace-otlp-http @opentelemetry/sdk-metrics @opentelemetry/exporter-metrics-otlp-http ``` -Same env vars. The TS exporter sends OTLP/HTTP JSON; that's fine. +Same env vars. ```ts import { setupTracer } from "@strands-agents/sdk/telemetry" @@ -183,14 +181,4 @@ Without Maple access (or with `MAPLE_TEST`), both must hold; silence alone prove ## Do not -- Do not omit `gen_ai_span_attributes_only`: Maple never reads span events, so the transcript would be empty. -- Do not set `OTEL_SEMCONV_STABILITY_OPT_IN` in code after an `Agent` exists. -- Do not create a second `TracerProvider` when one exists; do not call `StrandsTelemetry()` under `opentelemetry-instrument`/ADOT. -- Do not put a session id on a shared module-level agent. -- Do not rely on `session_manager` for Maple sessions; use `trace_attributes={"session.id": ...}`. -- Do not add another GenAI instrumentor (OpenLIT, OpenLLMetry, OpenInference, OpenAI/Bedrock instrumentation): duplicate model calls and tokens. - Do not also enable OpenRouter Broadcast (or another gateway trace export) for the same traffic: Strands `chat` spans have no `gen_ai.response.id`, so Maple can't dedupe and tokens double. -- Do not leave agents unnamed. -- Do not use `OTEL_TRACES_SAMPLER=traceidratio` unless the user wants it: dropped traces are dropped turns. -- Do not pin `strands-agents` below 1.51. -- Do not put PII in `trace_attributes`; it is never redacted. diff --git a/skills/maple-agent-tracing-vercel-ai-sdk/SKILL.md b/skills/maple-agent-tracing-vercel-ai-sdk/SKILL.md index f76dac337..876cb7635 100644 --- a/skills/maple-agent-tracing-vercel-ai-sdk/SKILL.md +++ b/skills/maple-agent-tracing-vercel-ai-sdk/SKILL.md @@ -5,14 +5,10 @@ description: "Trace Vercel AI SDK agents with Maple: register the AI SDK's OpenT # Maple agent tracing: Vercel AI SDK -Human guide with the reasoning: https://maple.dev/docs/agent-tracing/vercel-ai-sdk - ## Goal One conversation = one Maple Agent Session, one turn per `generate()`/`stream()` call, with the transcript, every model call (model, tokens, TTFT), every tool call (name, args, result, failures), and a lane per sub-agent. -How it works: AI SDK 7 emits GenAI-semconv spans through `@ai-sdk/otel` (`invoke_agent ` → `step ` → `chat ` + `execute_tool `) on tracer `gen_ai`, once `registerTelemetry(new OpenTelemetry())` has run. Content is on by default. Maple detects the AI SDK by its `gen_ai`/`ai` tracer scope and groups sessions by the conversation id you pass in `runtimeContext` (recorded as `ai.settings.context.conversationId` once `runtimeContext: true` is on). - Known gaps (tell the user, don't try to fix): cost shows as "unpriced" (AI SDK emits no cost; Maple never prices tokens); on AI SDK 5/6 the final assistant reply is missing from transcripts. If model calls go through OpenRouter, its Broadcast traces carry per-call cost and Maple matches them to the AI SDK `chat` spans by response id (see the OpenRouter guide). ## Step 0: Detect @@ -50,7 +46,7 @@ Use the repo's package manager (`@opentelemetry/api` arrives as a peer of `sdk-n OTEL_SERVICE_NAME=support-agent OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev -OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer +OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " ``` `NodeSDK()` with no span processors builds a batched OTLP http/protobuf exporter from these variables. Create `instrumentation.ts`: @@ -92,7 +88,6 @@ export const sdk = new NodeSDK({ spanProcessors: [spanProcessor] }) - `import "./instrumentation"` as the FIRST line of every entry point (server, worker, CLI). `registerTelemetry` must run before the first AI SDK call. - Keep `usage: true` (adds the reasoning-token breakdown, `ai.usage.*`) and `runtimeContext: true` (records the included runtime context keys; Maple reads the conversation id from them). - Do not pass `tracer:` to `OpenTelemetry` unless reusing a provider requires it; if you must, use `provider.getTracer("gen_ai")`. Other scope names break detection. -- Existing provider: add `spanProcessor` to it (`spanProcessors: [..., spanProcessor]` or the provider's add method) instead of creating `NodeSDK`. ## Step 2b: Install and init (Next.js) @@ -240,11 +235,4 @@ export class ConversationIdProcessor implements SpanProcessor { ## Do not - Do not use `experimental_telemetry: { isEnabled: true }` as the v7 setup; without `registerTelemetry` there are zero spans. -- Do not call `registerTelemetry` twice or register `LegacyOpenTelemetry` alongside `OpenTelemetry` (duplicate spans, double tokens). -- Do not start a second OpenTelemetry SDK next to an existing one; add a span processor. -- Do not use `telemetry.metadata` (removed in v7) or `ToolLoopAgent({ id })` for agent names. -- Do not stamp `maple_ai.session.id` on AI SDK spans: it re-vendors them and loses AI SDK decoding. - Do not add a second AI SDK tracer that also exports to Maple (Langfuse/Braintrust/Sentry AI integrations, OpenLLMetry/OpenInference AI SDK processors): every model call gets recorded twice. -- Do not return error payloads from failing tools; throw. -- Do not set attribute length limits. -- Do not use a gRPC exporter; Maple ingest is OTLP over HTTP. diff --git a/skills/maple-agent-tracing/SKILL.md b/skills/maple-agent-tracing/SKILL.md index 02cd44e2c..0d9e7529c 100644 --- a/skills/maple-agent-tracing/SKILL.md +++ b/skills/maple-agent-tracing/SKILL.md @@ -5,11 +5,11 @@ description: "Trace an AI agent or LLM app with Maple so each conversation shows # Maple agent tracing (router) -This skill only picks the right per-framework skill. Each framework has its own skill so you load the steps for your stack and nothing else. The human-readable overview is https://maple.dev/docs/agent-tracing. +This skill only picks the right per-framework skill. ## Step 1: Find every agent in the repo -Look for LLM and agent dependencies in every app and service: `package.json`, `pyproject.toml`, `requirements*.txt`, `uv.lock`, `pom.xml`, `build.gradle*`, `*.csproj`, `go.mod`. A repo can have more than one (a TypeScript chat backend and a Python worker, for example). Handle each one. +Look for LLM and agent dependencies in every app and service: `package.json`, `pyproject.toml`, `requirements*.txt`, `uv.lock`, `pom.xml`, `build.gradle*`, `*.csproj`, `go.mod`. A repo can have more than one; handle each one. ## Step 2: Install the matching skill and follow it