Observability

View as Markdown · Docs index for coding agents

The runtime exports your agents' runs as OpenTelemetry traces to any OTLP/HTTP endpoint: LangSmith, Langfuse, Honeycomb, Datadog, Grafana Tempo, Jaeger, or your own collector. Each run is a trace (or part of yours), with a span for every model call, tool call, compaction and wait for a person. Spans follow the OpenTelemetry GenAI semantic conventions, so GenAI-aware backends show them as LLM calls and tool calls.

curl -X PUT https://run.camelai.com/v1/telemetry \
  -H "Authorization: Bearer $CAMELAI_API_KEY" -H "Content-Type: application/json" \
  -d '{"endpoint": "https://api.honeycomb.io/v1/traces", "headers": {"x-honeycomb-team": "<key>"}}'

What your users and the model wrote (prompts, replies, tool arguments and results) is not exported unless you ask for it with include: {content: true}. See What is exported.

Setting it up

A PUT changes only the fields it carries: the others keep their current values (their defaults the first time). So {"include": {"content": true}} turns content on and leaves the endpoint, headers, protocol and sample rate as they were.

• GET /v1/telemetry shows the settings, header names, and status: when a node last exported, and why the last export failed (HTTP 401, Could not connect (ECONNREFUSED)), since the last success.
• POST /v1/telemetry/test sends one span (camelrun test span) now and answers with what your endpoint said and the span's traceId, to find it in your backend.
• DELETE /v1/telemetry stops export.

The SDKs, CLI and console do the same:

await runtime.telemetry.set({ endpoint: "https://api.honeycomb.io/v1/traces", headers: { "x-honeycomb-team": key } });
await runtime.telemetry.test();

await runtime.telemetry.set("https://api.honeycomb.io/v1/traces", headers={"x-honeycomb-team": key})
await runtime.telemetry.test()

camelrun telemetry set https://api.honeycomb.io/v1/traces --header x-honeycomb-team=@env:HONEYCOMB_KEY
camelrun telemetry test

In the console: Telemetry in the sidebar, with presets and a Send test span button.

See SDKs and CLI.

Presets

We checked these against each backend's documented OTLP endpoint and headers. LangSmith and Langfuse Cloud, Honeycomb and Datadog answer the runtime's test span at the URLs above (with a 401 or 403 until the key is real), and a self-hosted Langfuse (4.50), an OpenTelemetry Collector and Jaeger received runs in both encodings, nested as below.

A local collector or Jaeger

A hosted runtime cannot reach localhost. To look at traces on your machine, expose a local collector through a tunnel (cloudflared, ngrok) and give its HTTPS URL, or run a self-hosted runtime and allow the collector's origin: AGENT_OUTBOUND_ALLOW_ORIGINS=http://localhost:4318.

docker run --rm -p 16686:16686 -p 4318:4318 jaegertracing/all-in-one:latest
camelrun telemetry set http://localhost:4318      # a self-hosted runtime that allows that origin

Jaeger's UI is at localhost:16686, under the service camelrun.

Continuing your trace

Send a W3C traceparent header with a run, and its spans join your trace under the span you name:

POST /v1/agents/client_…/prompt
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

The SDKs pass it for you:

const run = await agent.run("Summarize ticket 123", { traceparent });

run = await agent.run("Summarize ticket 123", traceparent=traceparent)

With the OpenTelemetry API in your process, take the header from the active span (propagation.inject(context.active(), carrier) in TypeScript, opentelemetry.propagate.inject(carrier) in Python) and pass carrier["traceparent"]. The header is also taken on POST /v1/agents (a first prompt) and on the SDKs' POST /clients/:id/requests. It is not part of the request's idempotency: a retry under another span is the same run.

A run's record (GET /v1/agents/:id/requests/:requestId) carries trace: {traceId, spanId, parentSpanId?, sampled} when its tenant exports telemetry, so you can link from your own logs to the trace.

Spans

invoke_agent Support bot                      server   the run: accepted to ended
├─ chat claude-sonnet-4-5                     client   one model call (its retries are their own spans)
├─ execute_tool lookup_order                  internal a tool call
├─ chat claude-sonnet-4-5
├─ execute_tool js_exec                       internal model-written code
│  └─ execute_tool lookup_order               internal a tool call from that code
├─ compaction                                 client   summarizing the history to fit the context
└─ await_human_input                          internal a wait for a person (recorded when the run resumes)
invoke_agent Support bot                      server   the run that resumed after the input, under the run that asked

Each span has camelrun.tenant, camelrun.agent.id and camelrun.request.id.

The resource is service.name: camelrun. A failed span has status ERROR; its message is the error's class (rate_limit, context_overflow, tool_error) unless content is included, when it is the error's own message.

• A run that waits on a person ends with camelrun.run.status: input_required. When the input is answered, the run that resumes it is a new invoke_agent span in the same trace, under the run that asked, beside an await_human_input span for the wait (which can be days long).
• A compaction between runs belongs to no run: it is a trace of its own.
• A run that moves to another node (a deploy, a lost node) keeps its trace: each node exports the spans it made, the first node's cut-off calls marked error.type: interrupted, and the run's span, with the same id, comes from the node that finished it.
• Subagents. A tool call's span is the parent of whatever the call starts: a delegate's run continues the trace under the tool call that made it.

What is exported

By default, spans carry ids, names, models, token counts, costs, durations and outcomes: what you need to see what an agent did, how long it took and what it cost. The agent's name, your tools' names and the run's actor (your id for the user) are included; a message's metadata is not.

With include: {content: true} they also carry, each cut at 16,384 characters:

input.value and output.value repeat the GenAI attributes in the form LangSmith and Langfuse show as an observation's input and output. A model call's input (the whole context it was sent) is not exported: its new messages are the prompt and the tool results before it.

Content goes only to your endpoint; the runtime's own logs never hold it. Turn it on for development, or where your backend is allowed to keep your users' data.

Delivery

Export never slows a run. Each node queues the spans it makes in memory and sends them in batches (up to 512 spans, about every 2 seconds), retrying 429, 502, 503, 504 and connection failures with backoff (honoring Retry-After) up to 5 attempts. Other answers drop the batch. A node keeps at most 5,000 waiting spans per tenant and 20,000 in all; spans beyond those, and batches that run out of attempts, are dropped and counted. Export is best effort: spans are kept in memory, not durably, and a node that stops sends what it can within a few seconds. A batch your endpoint accepted after its 10-second timeout is sent again, so a span can arrive twice; backends keep one per span id.

Endpoints the runtime refuses

The endpoint goes through the same guard as webhooks and MCP servers: HTTPS only, no credentials in the URL, and no private, loopback, link-local (including cloud metadata addresses) or reserved addresses, checked again at each connection (400 when it is set; Could not connect in status.lastError if a name later resolves to one). Redirects are not followed. Self-hosted runtimes can allow an internal collector with AGENT_OUTBOUND_ALLOW_ORIGINS.