# OTEL spans & exporter wiring (/docs/otel-observability) OTel spans are one surface in the **observability** [thematic island](/docs/concept-clusters#what-lives-outside-the-cluster-pattern) — siblings of [observability](/docs/observability) (the orientation page), [lineage](/docs/lineage), and [runtime inspector](/docs/runtime-inspector). This page is the read-side OTel surface. For the write-side audit ledger that the `llm.invocation` span pairs one-to-one with, see [Audit ledger](/docs/audit-ledger). This page covers the four span types the substrate emits, the parent-threading rules that nest them correctly, the auto-flush knob, the `runtime.spans` in-process introspection facet, and a minimal OTLP exporter wiring. For non-OTel observability — structured logs, Prometheus, Datadog statsd, lifecycle-event streaming — see [Observability](/docs/observability). ## The four span types [#the-four-span-types] The substrate emits exactly four span types. Every other span in your trace tree is either a parent (your HTTP handler) or a child that you wired yourself. | Span type | Where emitted | Key attributes | | ---------------- | ---------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | `session.turn` | Around `runtime.executeMessage` for one user turn | `pleach.session_id`, `pleach.turn_id`, `pleach.tenant_id`, `pleach.actor_kind` | | `llm.invocation` | One per LLM call — same cardinality as the audit ledger row | `pleach.model_id`, `pleach.family`, `pleach.call_class`, `pleach.tokens.input`, `pleach.tokens.output`, `pleach.tenant_id` | | `graph.stage` | Around each orchestrator stage transition | `pleach.stage_id` (`anchor-plan` / `tool-loop` / `synthesize` / `post-turn`), `pleach.session_id`, `pleach.turn_id` | | `tool.execution` | Per tool dispatch — pairs with the `tool.completed` stream event | `pleach.tool_name`, `pleach.capability_id`, `pleach.status`, `pleach.duration_ms` | These four are the cardinality-correct seams. Adding your own per-call span at a custom seam double-counts against `llm.invocation` and clutters the trace tree. ## Parent-threading [#parent-threading] Spans nest naturally: a `session.turn` parents the `graph.stage`s fired during that turn; each `graph.stage` parents the `llm.invocation`s and `tool.execution`s emitted during that stage. The substrate threads the active span through the OTel context API (`context.with(setSpan(ctx, span), fn)`), so consumer spans created inside a runtime callback nest under the active stage without manual parenting. If your HTTP handler opens its own outer span, `session.turn` nests under that — the chain reads `http.request → session.turn → graph.stage → llm.invocation` end to end. ## Auto-flush [#auto-flush] The exporter buffer flushes on a turn cadence. The knob lives on `SessionRuntimeConfig`: ```typescript import { SessionRuntime } from "@pleach/core"; const runtime = new SessionRuntime({ storage: memoryAdapter, userId: "user-123", otelFlushIntervalTurns: 1, // default: flush every turn }); ``` Default is `1` — the buffer drains at the end of every `session.turn`. Raise it (`5`, `10`) to batch in high-throughput deployments where the per-turn flush cost shows up in tail latency. Set it to `0` to disable the runtime-driven flush entirely and let the SDK's batch processor drive cadence on its own timer. ## `runtime.spans` facet [#runtimespans-facet] The `runtime.spans` facet is the in-process handle on the wired exporter. It exposes the exporter lifecycle — `start(input)` to open a host span, `flush()` to drain buffered snapshots, `shutdown()` to close the exporter — plus three introspection reads: ```typescript import { SessionRuntime } from "@pleach/core"; const runtime = new SessionRuntime({ /* ... */ }); await runtime.executeMessage(sessionId, "hello"); runtime.spans.inFlightCount(); // number — spans started but not yet ended runtime.spans.isShutdown(); // boolean — has the exporter been shut down? runtime.spans.snapshot(); // → { inFlightCount: number; isShutdown: boolean } ``` `snapshot()` returns the exporter's lifecycle state, **not** the span records — it's `{ inFlightCount, isShutdown }`, a single read of the two introspection counters. Calling `.find(...)` on it throws; it is not an array. The default `NoopOtelExporter` tracks both counters accurately even when no sink is wired. To READ the emitted span records in-process — for test assertions or a DevTools trace-tree panel — wire the reference `CapturingOtelExporter` and read its `captured` array directly. It captures every span snapshot in memory instead of forwarding to an external collector: ```typescript import { SessionRuntime } from "@pleach/core"; import { CapturingOtelExporter } from "@pleach/core/otel"; const exporter = new CapturingOtelExporter(); const runtime = new SessionRuntime({ /* ... */, otelExporter: exporter }); await runtime.executeMessage(sessionId, "hello"); const turnSpan = exporter.captured.find((s) => s.name === "session.turn"); // each captured span is a PleachSpanSnapshot: // { // name: string; // kind: "session.turn" | "llm.invocation" | "graph.stage" // | "tool.execution" | "host.custom"; // id: string; // parentId: string | undefined; // attributes: Record; // includes the pleach.* keys // status: "ok" | "error" | "unset"; // error: Error | undefined; // startTimeNs: bigint; // endTimeNs: bigint; // durationNs: bigint; // } ``` `CapturingOtelExporter` is the test/DevTools sink. Production hosts wire a real sink adapter instead — see the recipe below. ## Exporter substrate and a minimal OTLP recipe [#exporter-substrate-and-a-minimal-otlp-recipe] `@pleach/core` has zero `@opentelemetry/*` imports. It emits its four span types through one contract — `OtelExporter` — and nothing else. The runtime does **not** read the global tracer provider. If `config.otelExporter` is omitted, the runtime uses the default `NoopOtelExporter`, which drops every span. A NodeSDK started alongside the runtime captures no pleach spans on its own. So a real OTLP pipeline needs two pieces: the OTel SDK (configures the destination and registers a tracer provider) **and** a small bridge that implements `OtelExporter` against `@opentelemetry/api`, passed to the runtime as `config.otelExporter`. The bridge lives in your host code (or a `@pleach/observability` adapter) — it's the only part that imports the OTel SDK. ```typescript import { NodeSDK } from "@opentelemetry/sdk-node"; import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"; import { trace, SpanStatusCode } from "@opentelemetry/api"; import { SessionRuntime } from "@pleach/core"; import type { OtelExporter, PleachSpanStart, PleachSpanHandle, } from "@pleach/core/otel"; // 1. Configure and start the OTel SDK. This registers a tracer // provider and points the OTLP exporter at your collector. const sdk = new NodeSDK({ traceExporter: new OTLPTraceExporter({ url: process.env.OTLP_ENDPOINT, // e.g. https://api.honeycomb.io/v1/traces headers: { "x-honeycomb-team": process.env.HONEYCOMB_API_KEY ?? "", }, }), }); sdk.start(); // 2. Bridge @pleach/core's OtelExporter contract to @opentelemetry/api. // This is the only file that imports the OTel SDK; core never does. const tracer = trace.getTracer("@pleach/core"); class OtelApiExporter implements OtelExporter { startSpan(start: PleachSpanStart): PleachSpanHandle { const span = tracer.startSpan(start.name, { attributes: { ...start.attributes }, // carries pleach.tenant_id, pleach.family, … }); return { id: crypto.randomUUID(), setAttribute: (k, v) => span.setAttribute(k, v as never), setAttributes: (attrs) => span.setAttributes(attrs as never), setStatus: (status, error) => span.setStatus({ code: status === "error" ? SpanStatusCode.ERROR : SpanStatusCode.OK, message: error?.message, }), recordError: (error) => { span.recordException(error); span.setStatus({ code: SpanStatusCode.ERROR }); }, end: () => span.end(), }; } async flush(): Promise { /* the SDK's batch processor drains on its own timer */ } async shutdown(): Promise { await sdk.shutdown(); } } // 3. Wire the bridge into the runtime. WITHOUT this, spans fall to the // default NoopOtelExporter and go nowhere. const runtime = new SessionRuntime({ storage: memoryAdapter, userId: "user-123", otelExporter: new OtelApiExporter(), }); ``` The bridge is destination-agnostic: the same `OtelApiExporter` lands in Honeycomb, Datadog's OTLP intake, Grafana Tempo, and any self-hosted OpenTelemetry Collector — only the OTLP `url` and auth headers in step 1 differ. For Datadog APM, point `OTLP_ENDPOINT` at the agent's OTLP intake (`http://localhost:4318/v1/traces` by default). For Tempo or a self-hosted Collector, point at the collector's OTLP HTTP receiver. ## `pleach.tenant_id` attribute [#pleachtenant_id-attribute] Every span carries `pleach.tenant_id` when `runtime.tenant` is configured, or when an outbound HTTP call rides through the `withTenantHeader` adapter. The attribute is set at span-start from the active tenant context, so it appears on all four span types without per-call wiring on your side. This is the load-bearing field for per-tenant trace queries — the Honeycomb / Tempo / Datadog query `WHERE pleach.tenant_id = $1` gives you a tenant's full trace history without joining to the audit ledger. See [Tenant facet](/docs/tenant-facet). ## Audit gate [#audit-gate] `audit:otel-noop-soak` runs in CI to catch span-emit drift. It asserts two things: 1. The substrate emits exactly one of each of the four span types when a turn runs against an in-process exporter. A refactor that silently stops emitting one (a missing `tracer.startSpan` after a control-flow change) fails the gate. 2. The noop default — no exporter wired, no SDK started — stays genuinely noop. No allocations from span construction, no buffered records hanging on a process-global. For consumers, the gate is informational: it tells you the substrate's emit contract is enforced upstream. You don't run it; the upstream CI does, and a release that ships with it red is the release that breaks your trace tree. ## What not to do [#what-not-to-do] * Don't add wall-clock timestamps as span attributes. Spans carry their own timing via `startTimeUnixNano` / `endTimeUnixNano`; a second clock disagrees with the first about subtle things (NTP skew, monotonic vs wall). * Don't emit your own per-call span at a custom seam. The `llm.invocation` span is the cardinality-correct one — one per audit row, one per actual provider call. A custom span at a provider-wrapper site double-counts; a span in a stream observer fires per chunk and floods the trace tree. * Don't share a single OTLP exporter across worker processes without configuring the SDK for it. The default batch processor isn't fork-safe; each worker needs its own `NodeSDK.start()` after the fork, or the parent's exporter ends up holding spans the workers emit. * Don't disable `pleach.tenant_id` to "reduce cardinality." It's the join key every tenant-scoped trace query reads. Cardinality control belongs at the exporter's sampler, not at the attribute source. ## Where to go next [#where-to-go-next]