Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 6 additions & 23 deletions skills/maple-agent-tracing-agno/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,7 @@ description: "Trace Agno agents with Maple: installs the OpenInference Agno inst

Goal: every conversation with the Agno app shows up in Maple **Agent Sessions** as exactly one session, one turn per `run()`, with transcript, model calls, tool calls (failures marked), team-member lanes, tokens and cost where the provider returns it.

Human guide with the reasoning: https://maple.dev/docs/agent-tracing/agno

Mechanism: `openinference-instrumentation-agno` (the instrumentor Agno's own `setup_tracing()` uses) + OTel SDK + OTLP/HTTP exporter to Maple. Agno's `setup_tracing(db=...)` and `AgentOS(tracing=True)` only write to the AgentOS database; they never export OTLP.
Mechanism: `openinference-instrumentation-agno` + OTel SDK + OTLP/HTTP exporter to Maple. Agno's `setup_tracing(db=...)` and `AgentOS(tracing=True)` only write to the AgentOS database; they never export OTLP.

## Step 0: Detect versions and existing setup

Expand Down Expand Up @@ -52,12 +50,12 @@ opentelemetry-exporter-otlp-proto-http
OTEL_SERVICE_NAME=<service name, e.g. support-agent>
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=<env>
OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev # EU: https://ingest.eu.maple.dev
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer <key>
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <key>"
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
AGNO_TELEMETRY=false
```

The exporter appends `/v1/traces` to `OTEL_EXPORTER_OTLP_ENDPOINT`. If you use `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` instead, give the full `.../v1/traces` URL. `AGNO_TELEMETRY=false` disables Agno's anonymous usage pings (unrelated to OTel).
The exporter appends `/v1/traces` to `OTEL_EXPORTER_OTLP_ENDPOINT`. If you use `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` instead, give the full `.../v1/traces` URL.

### 2c. Tracing module

Expand All @@ -81,7 +79,7 @@ AgnoInstrumentor().instrument(
)
```

- `enable_genai_semconv=True` is REQUIRED. It dual-writes `gen_ai.*` (messages in `{role, parts}` form, usage, tool name/args/result, agent name, `gen_ai.operation.name`). Env equivalent: `OPENINFERENCE_ENABLE_GENAI_SEMCONV=true` (use it when you can't edit the `instrument()` call, e.g. AgentOS owns it).
- `enable_genai_semconv=True` is REQUIRED. Env equivalent: `OPENINFERENCE_ENABLE_GENAI_SEMCONV=true` (use it when you can't edit the `instrument()` call, e.g. AgentOS owns it).
- Import `tracing` at the top of the entry point (`main.py`, `app.py`, the ASGI module), before building agents, teams or `AgentOS`.
- AgentOS: `AgentOS(tracing=True)` and `setup_tracing(db=...)` skip their setup when a real `TracerProvider` is already registered, so AgentOS's traces view stops getting spans once `tracing.py` runs first. If the user wants to keep that view, add Agno's DB exporter to the same provider (use the `db` the AgentOS uses):

Expand All @@ -94,8 +92,6 @@ AgnoInstrumentor().instrument(

## Step 3: Session id (one conversation = one session)

Maple groups Agno traces by `session.id` on the run span. Agno sets it from `session_id=`.

- Pass `session_id=<conversation id>` on EVERY `run`, `arun`, `print_response`, `aprint_response`, `continue_run`, `acontinue_run`, and on `team.run/arun` and `workflow.run/arun`. Use the id the app already stores for the chat/thread; pass `user_id=` too if available.
- Without `session_id=`, Agno mints a `uuid4()` on the first run and stores it on the `Agent`/`Team` instance; every later run of that instance reuses it. A module-level shared agent then merges all users into one session. Fix it; don't rely on the default.
- Do not mint a new id per request (every turn becomes its own session).
Expand Down Expand Up @@ -180,27 +176,14 @@ Raw span check: run spans `<agent_name>.run` have `session.id` + `gen_ai.operati
## Known behaviours (expected; explain if the user asks)

- If `setup_tracing()`/`AgentOS(tracing=True)` ran first, `set_tracer_provider` in `tracing.py` logs "Overriding of current TracerProvider is not allowed" and spans go only to the AgentOS database. A second `instrument()` call logs "Attempting to instrument while already instrumented" and its `config` is ignored.
- The instrumentor patches Agno's run functions and every model class in `agno.models`, so agents created after `tracing.py` runs are traced with no further changes.
- Without `enable_genai_semconv`, spans carry only OpenInference attributes (`llm.input_messages.0.message.content`, `llm.token_count.prompt`).
- Maple reads span attributes only; the instrumentor emits no span events or OTLP logs, so nothing else needs enabling. Masked values are replaced with `__REDACTED__` in-process before export.
- Team trace shape (one trace per team run): member runs sit directly under the leader's run, next to (not inside) the `delegate_task_to_member` tool spans; those show as ordinary tool calls on the leader with member id and task as arguments. With `team.arun()` in `coordinate` mode, members called in one step run concurrently and their spans overlap; sync `team.run()` runs them sequentially.
- Teams: `delegate_task_to_member` tool spans show as ordinary tool calls on the leader with member id and task as arguments. With `team.arun()` in `coordinate` mode, members called in one step run concurrently and their spans overlap; sync `team.run()` runs them sequentially.
- Team context leak is upstream issue agno#5573. "Failed to detach context" in the logs after a streamed team run is agno#5208: log noise, spans still export.
- Human-in-the-loop: the paused run (`<agent>.run`) and the resumed run (`<agent>.continue_run`) are two traces and two turns in one session; the approved tool call appears once, in the second. Instrumentor <1.0.8 doesn't wrap `continue_run()`, so resumed runs appear as loose model/tool calls with no session.
- Tokens: every model span has input/output tokens plus cache read/write when the provider reports them; streamed runs record usage from the final chunk. The run span has no tokens, so nothing is double counted. Reasoning tokens aren't broken out. Maple links tool results to calls through the message history since tool spans lack `gen_ai.tool.call.id`, and can't show streaming latency (no TTFT).
- Cost: OpenRouter's price is recorded as `llm.cost.total` (USD, instrumentor >=1.0.10). Providers called directly (OpenAI, Anthropic) return no price, so sessions are unpriced.
- A tool that raises: Agno catches it and hands the message to the model; the tool span is `ERROR`, the run span stays OK. Maple groups repeated failures by the exception message.
- A tool that raises: Agno catches it and hands the message to the model; the tool span is `ERROR`, the run span stays OK.

## Do not

- Do not rely on `setup_tracing()` or `AgentOS(tracing=True)` to reach Maple; they only write to the database.
- Do not omit `TraceConfig(enable_genai_semconv=True)` / `OPENINFERENCE_ENABLE_GENAI_SEMCONV=true`.
- Do not call `AgnoInstrumentor().instrument()` twice or create a second `TracerProvider`.
- Do not stack another LLM instrumentor (OpenAI/LiteLLM OpenInference, OpenLIT, `auto_instrument=True`) on the same calls.
- Do not run agents without `session_id=` in a server; do not generate a fresh id per request.
- Do not put a different `session_id` on team members or resumed runs than on the conversation.
- Do not stamp `maple_ai.session.id` on Agno spans: it re-vendors them and loses decoding. Agno's own `session.id` is what Maple reads.
- Do not use a console/stdout exporter in production, and do not use `SimpleSpanProcessor` in servers (it exports synchronously on the request path).
- Do not skip the flush in scripts, notebooks, CLIs and serverless.
- Do not run an agent after a team run in the same thread/task without isolating the team run (Step 5).
- Do not expect `gen_ai.response.id`, `gen_ai.tool.call.id` on tool spans, TTFT or reasoning-token attributes from this instrumentor; they are not emitted and nothing needs fixing.
- Do not set `OTEL_SDK_DISABLED=true` (it turns off all tracing). `AGNO_TELEMETRY=false` only stops Agno's product analytics and is safe.
18 changes: 3 additions & 15 deletions skills/maple-agent-tracing-claude-agent-sdk/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,11 @@ description: "Trace Claude Agent SDK agents (TypeScript and Python) and Claude C

# Maple agent tracing: Claude Agent SDK and Claude Code

Human guide with the reasoning: https://maple.dev/docs/agent-tracing/claude-agent-sdk

## Goal

One conversation = one Maple Agent Session, one turn per user message, with the user prompts, every model call (model, tokens, TTFT), every tool call (name, args, result, failures).

How it works: the Agent SDK emits nothing itself. `query()` spawns the Claude Code CLI, which has OpenTelemetry built in and exports spans `claude_code.interaction` (turn), `claude_code.llm_request` (model call), `claude_code.tool` (tool call), with phase children `claude_code.tool.blocked_on_user` / `claude_code.tool.execution`. All configuration is environment variables for that child process. Maple detects the `com.anthropic.claude_code*` scopes (spans come from `com.anthropic.claude_code.tracing`), groups by the `session.id` span attribute, and restates the spans as `gen_ai.*` at ingest. No instrumentation package, no TracerProvider.
How it works: the Agent SDK emits nothing itself. `query()` spawns the Claude Code CLI, which has OpenTelemetry built in and exports spans `claude_code.interaction` (turn), `claude_code.llm_request` (model call), `claude_code.tool` (tool call), with phase children `claude_code.tool.blocked_on_user` / `claude_code.tool.execution`. All configuration is environment variables for that child process. No instrumentation package, no TracerProvider.

Known gaps (tell the user, don't try to fix): assistant reply text and cost are only on OTLP log events, which Maple's session views don't read, so transcripts have no assistant text and sessions show "unpriced"; no `gen_ai.agent.name`, so sub-agents get no separate lanes; tool arguments shown only for Bash (command) and Read/Edit/Write (file path).

Expand Down Expand Up @@ -234,15 +232,5 @@ If spans exist but the turn nests under an unrelated trace, an inherited `TRACEP

## Do not

- Do not omit `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1`: zero spans without it (metrics/logs still flow, which hides the problem).
- Do not leave `OTEL_EXPORTER_OTLP_PROTOCOL` unset or use `grpc`: Claude Code has no default; Maple ingest is OTLP/HTTP.
- Do not set any exporter to `console` in SDK apps: stdout is the SDK message channel.
- Do not pass a TS `env` without `...process.env`.
- Do not pass an inherited `TRACEPARENT`/`TRACESTATE` to the CLI (Claude Code's Bash tool and CI set them).
- Do not call `query()` per message without `resume`.
- Do not put telemetry vars in a repo's `.claude/settings.json`.
- Do not assume `options.env` wins: `env` in loaded settings files overrides it (Step 0).
- Do not add a second instrumentation (OpenInference/Langfuse/LangSmith hooks) on top of the CLI's spans.
- Do not use `OTEL_METRICS_INCLUDE_SESSION_ID=false`, `forkSession` on normal turns, or detailed beta tracing.
- Do not promise cost or assistant replies in Maple's session views; they are in Logs (`claude_code.api_request` `cost_usd`, `claude_code.assistant_response`) and the `claude_code.cost.usage` metric.
- Do not print or commit `maple_sk_` keys or model API keys.
- Do not omit `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1`: zero spans, while metrics/logs still flow and hide it.
- Do not leave `OTEL_EXPORTER_OTLP_PROTOCOL` unset or `grpc`: Claude Code has no default; Maple ingest is OTLP/HTTP.
13 changes: 0 additions & 13 deletions skills/maple-agent-tracing-cloudflare-agents/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,10 @@ description: "Trace Cloudflare Agents SDK agents (AIChatAgent, Agent on Durable

# Maple agent tracing: Cloudflare Agents SDK

Human guide: https://maple.dev/docs/agent-tracing/cloudflare-agents

## Goal

One chat (one agent instance) = one Maple Agent Session, one turn per user message, with the transcript, every model call (model, tokens), every tool call (name, args, result, failures), and a lane per sub-agent.

How it works: the Agents SDK does not emit GenAI spans itself. Model calls go through the Vercel AI SDK, whose `@ai-sdk/otel` integration emits `invoke_agent <model>` → `step <n>` → `chat <model>` + `execute_tool <tool>` on tracer scope `gen_ai`. Maple detects them as **Vercel AI SDK** (by the `gen_ai` tracer scope) and groups sessions by the conversation id you pass in `runtimeContext` (recorded as `ai.settings.context.conversationId` once `runtimeContext: true` is on). Workers can't run `@opentelemetry/sdk-node`, so you build a `BasicTracerProvider` with the OTLP HTTP exporter (its browser/worker build posts OTLP JSON with `fetch`) and call `forceFlush()` at the end of each turn.

Verified end to end (wrangler dev, workerd 1.20260926.1, a local OTLP receiver, `MockLanguageModelV4` from `ai/test`) with: `agents` 0.24.0, `@cloudflare/ai-chat` 0.12.0, `ai` 7.0.122, `@ai-sdk/otel` 1.0.122, `wrangler` 4.143.0, `@opentelemetry/sdk-trace-base` / `resources` / `context-async-hooks` 2.11.0, `@opentelemetry/exporter-trace-otlp-http` 0.222.0, `@opentelemetry/api` 1.9.1. Both `AIChatAgent` over its WebSocket protocol (2 turns x 2 chats, streamed, tool call, sub-agent) and a plain `Agent.onRequest` with `generateText`.

Known gaps (tell the user, don't try to fix): cost shows as "unpriced" (the AI SDK emits no cost). These traces are separate from Cloudflare's native Workers traces (different trace ids); that's expected.

## Step 0: Detect
Expand Down Expand Up @@ -169,11 +163,4 @@ Then against Maple: the `wrangler dev` output shows no export errors (the diag l

## Do not

- Do not use `@opentelemetry/sdk-node` / `NodeSDK` / `@vercel/otel` in a Worker.
- Do not rely on Cloudflare's native trace destinations or `cloudflare:workers` `tracing` for the agent spans; they don't carry AI SDK spans.
- Do not skip the per-turn flush.
- Do not call `registerTelemetry` twice or register `LegacyOpenTelemetry` next to `OpenTelemetry` (duplicate spans, double tokens).
- Do not use a per-request id, `ctx.id.toString()` of a shared DO, or a module constant as the conversation id.
- Do not stamp `maple_ai.session.id` on AI SDK spans: it re-vendors them and loses AI SDK decoding.
- Do not add a second AI SDK tracer that also exports to Maple (Langfuse/Braintrust/Sentry AI integrations): every model call gets recorded twice.
- Do not put the ingest key in `vars` in the Wrangler config or in client code; use a secret.
17 changes: 4 additions & 13 deletions skills/maple-agent-tracing-crewai/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ description: "Trace CrewAI crews and flows with Maple: OpenInference CrewAI inst

# Maple agent tracing: CrewAI

Goal: every conversation the app runs through CrewAI shows up in Maple **Agent Sessions** as ONE session, with the transcript, each model call (model, tokens), each tool call (name, result, failure) and one lane per agent role. Reasoning and background for every step: https://maple.dev/docs/agent-tracing/crewai
Goal: every conversation the app runs through CrewAI shows up in Maple **Agent Sessions** as ONE session, with the transcript, each model call (model, tokens), each tool call (name, result, failure) and one lane per agent role.

CrewAI exports nothing to your backend. Its built-in telemetry is anonymous analytics to crewai.com on a private provider; its OTel export is AMP-only. All spans come from OpenInference, and the defaults are wrong for Maple in three ways this skill fixes: the CrewAI instrumentor records no model calls (a second, SDK-level instrumentor is required), no session id, and no agent-name attribute.
CrewAI exports nothing to your backend. All spans come from OpenInference, and the defaults are wrong for Maple in three ways this skill fixes: the CrewAI instrumentor records no model calls (a second, SDK-level instrumentor is required), no session id, and no agent-name attribute.

## Step 0: Detect versions and existing OpenTelemetry

Expand Down Expand Up @@ -209,25 +209,16 @@ Otherwise check in Maple **Agent Sessions** (`https://app.maple.dev/agent-sessio
- Cost: unpriced unless models go through LiteLLM (which records `llm.cost.total`). Expected.
- No spans with scope `crewai.telemetry`, no `coding_agent` attribute (telemetry is off).

More details (from the human guide, for edge cases):
Edge cases:

- Tokens: model spans carry `gen_ai.usage.input_tokens`/`output_tokens` (plus cached and reasoning tokens when reported); crew and agent spans carry none, so nothing double-counts. CrewAI's OpenAI provider always requests `stream_options={"include_usage": True}` when streaming, so streamed calls keep tokens.
- Model = the one the provider returned (e.g. `anthropic/claude-haiku-4.5` behind OpenRouter); provider = the SDK used, so every OpenRouter model shows `openai`.
- `memory=True` / `planning=True` add real, billed model calls (memory analysis, embeddings, planning agent); they appear in the session. Expected.
- The Arize Phoenix CrewAI page still recommends the LiteLLM instrumentor; ignore it for native providers (it records nothing there).
- Flow span layout: `<flow name>.kickoff` root, one `<flow name>.<method>` span per `@start`/`@listen`/`@router` method, crews and `Agent.kickoff()` nested inside. Conversational flow turns show `<flow>.route_conversation` and `<flow>.converse_turn` under the kickoff.
- Why the streaming wrapper works: `gen_ai.operation.name=invoke_agent` makes Maple treat it as the turn's agent span, so the two crew kickoff spans under it are one turn.

If sessions are split per message: `using_session` missing or id changing. No model spans/tokens: wrong or missing SDK instrumentor. Every call its own trace: `akickoff`. Nothing arrives: exporter endpoint/header wrong, `.env` loaded after `tracing.py`, `OTEL_SDK_DISABLED=true`, or process exited without flushing. 401 `ingest_unauthorized` / "Invalid ingest key" with a key you trust: keys are region-bound, so it likely belongs to the other region; try the other endpoint.

## Do not

- Do not install only `openinference-instrumentation-crewai`: no model calls, no tokens, no transcript.
- Do not add the LiteLLM instrumentor for `openai/`/`openrouter/`/`anthropic/`/`gemini/` models (CrewAI 1.x calls those SDKs natively; LiteLLM records nothing). Do not stack two model-layer instrumentors or `litellm.callbacks=["otel"]` on the same calls (duplicate spans).
- Do not use `OTEL_SDK_DISABLED=true` to silence CrewAI telemetry.
- Do not use `crewai.telemetry`, `share_crew`, or `CREWAI_TRACING_ENABLED=true` as the Maple pipeline.
- Do not use `crew_id`, `crew_key`, `task_id` or a fresh UUID per request as the session id.
- Do not call `akickoff()` on traced crews.
- Do not pass `OTLPSpanExporter(endpoint="https://ingest.maple.dev")` without `/v1/traces`.
- Do not create a second `TracerProvider` when one exists, and do not call `instrument()` twice.
- Do not wrap tools in try/except that returns error strings; let them raise.
- Do not stack two model-layer instrumentors (or `litellm.callbacks=["otel"]`) on the same calls: duplicate model spans and tokens.
Loading
Loading