OTEL spans & exporter wiring

OTel spans are one surface in the observability thematic island — siblings of observability (the orientation page), lineage, and runtime inspector.

This page is the read-side OTel surface. For the write-side audit ledger that the llm.invocation span pairs one-to-one with, see Audit ledger.

This page covers the four span types the substrate emits, the parent-threading rules that nest them correctly, the auto-flush knob, the runtime.spans in-process introspection facet, and a minimal OTLP exporter wiring. For non-OTel observability — structured logs, Prometheus, Datadog statsd, lifecycle-event streaming — see Observability.

The four span types

The substrate emits exactly four span types. Every other span in your trace tree is either a parent (your HTTP handler) or a child that you wired yourself.

These four are the cardinality-correct seams. Adding your own per-call span at a custom seam double-counts against llm.invocation and clutters the trace tree.

Parent-threading

Spans nest naturally: a session.turn parents the graph.stages fired during that turn; each graph.stage parents the llm.invocations and tool.executions emitted during that stage.

The substrate threads the active span through the OTel context API (context.with(setSpan(ctx, span), fn)), so consumer spans created inside a runtime callback nest under the active stage without manual parenting. If your HTTP handler opens its own outer span, session.turn nests under that — the chain reads http.request → session.turn → graph.stage → llm.invocation end to end.

Auto-flush

The exporter buffer flushes on a turn cadence. The knob lives on SessionRuntimeConfig:

import { SessionRuntime } from "@pleach/core";

const runtime = new SessionRuntime({
  storage: memoryAdapter,
  userId: "user-123",
  otelFlushIntervalTurns: 1,   // default: flush every turn
});

Default is 1 — the buffer drains at the end of every session.turn. Raise it (5, 10) to batch in high-throughput deployments where the per-turn flush cost shows up in tail latency. Set it to 0 to disable the runtime-driven flush entirely and let the SDK's batch processor drive cadence on its own timer.

runtime.spans facet

The runtime.spans facet is the in-process handle on the wired exporter. It exposes the exporter lifecycle — start(input) to open a host span, flush() to drain buffered snapshots, shutdown() to close the exporter — plus three introspection reads:

import { SessionRuntime } from "@pleach/core";

const runtime = new SessionRuntime({ /* ... */ });
await runtime.executeMessage(sessionId, "hello");

runtime.spans.inFlightCount();   // number — spans started but not yet ended
runtime.spans.isShutdown();      // boolean — has the exporter been shut down?
runtime.spans.snapshot();        // → { inFlightCount: number; isShutdown: boolean }

snapshot() returns the exporter's lifecycle state, not the span records — it's { inFlightCount, isShutdown }, a single read of the two introspection counters. Calling .find(...) on it throws; it is not an array. The default NoopOtelExporter tracks both counters accurately even when no sink is wired.

To READ the emitted span records in-process — for test assertions or a DevTools trace-tree panel — wire the reference CapturingOtelExporter and read its captured array directly. It captures every span snapshot in memory instead of forwarding to an external collector:

import { SessionRuntime } from "@pleach/core";
import { CapturingOtelExporter } from "@pleach/core/otel";

const exporter = new CapturingOtelExporter();
const runtime = new SessionRuntime({ /* ... */, otelExporter: exporter });
await runtime.executeMessage(sessionId, "hello");

const turnSpan = exporter.captured.find((s) => s.name === "session.turn");
// each captured span is a PleachSpanSnapshot:
// {
//   name: string;
//   kind: "session.turn" | "llm.invocation" | "graph.stage"
//       | "tool.execution" | "host.custom";
//   id: string;
//   parentId: string | undefined;
//   attributes: Record<string, unknown>;   // includes the pleach.* keys
//   status: "ok" | "error" | "unset";
//   error: Error | undefined;
//   startTimeNs: bigint;
//   endTimeNs: bigint;
//   durationNs: bigint;
// }

CapturingOtelExporter is the test/DevTools sink. Production hosts wire a real sink adapter instead — see the recipe below.

Exporter substrate and a minimal OTLP recipe

@pleach/core has zero @opentelemetry/* imports. It emits its four span types through one contract — OtelExporter — and nothing else. The runtime does not read the global tracer provider. If config.otelExporter is omitted, the runtime uses the default NoopOtelExporter, which drops every span. A NodeSDK started alongside the runtime captures no pleach spans on its own.

So a real OTLP pipeline needs two pieces: the OTel SDK (configures the destination and registers a tracer provider) and a small bridge that implements OtelExporter against @opentelemetry/api, passed to the runtime as config.otelExporter. The bridge lives in your host code (or a @pleach/observability adapter) — it's the only part that imports the OTel SDK.

import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { trace, SpanStatusCode } from "@opentelemetry/api";
import { SessionRuntime } from "@pleach/core";
import type {
  OtelExporter,
  PleachSpanStart,
  PleachSpanHandle,
} from "@pleach/core/otel";

// 1. Configure and start the OTel SDK. This registers a tracer
//    provider and points the OTLP exporter at your collector.
const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTLP_ENDPOINT,                // e.g. https://api.honeycomb.io/v1/traces
    headers: {
      "x-honeycomb-team": process.env.HONEYCOMB_API_KEY ?? "",
    },
  }),
});
sdk.start();

// 2. Bridge @pleach/core's OtelExporter contract to @opentelemetry/api.
//    This is the only file that imports the OTel SDK; core never does.
const tracer = trace.getTracer("@pleach/core");

class OtelApiExporter implements OtelExporter {
  startSpan(start: PleachSpanStart): PleachSpanHandle {
    const span = tracer.startSpan(start.name, {
      attributes: { ...start.attributes },        // carries pleach.tenant_id, pleach.family, …
    });
    return {
      id: crypto.randomUUID(),
      setAttribute: (k, v) => span.setAttribute(k, v as never),
      setAttributes: (attrs) => span.setAttributes(attrs as never),
      setStatus: (status, error) =>
        span.setStatus({
          code: status === "error" ? SpanStatusCode.ERROR : SpanStatusCode.OK,
          message: error?.message,
        }),
      recordError: (error) => {
        span.recordException(error);
        span.setStatus({ code: SpanStatusCode.ERROR });
      },
      end: () => span.end(),
    };
  }
  async flush(): Promise<void> {
    /* the SDK's batch processor drains on its own timer */
  }
  async shutdown(): Promise<void> {
    await sdk.shutdown();
  }
}

// 3. Wire the bridge into the runtime. WITHOUT this, spans fall to the
//    default NoopOtelExporter and go nowhere.
const runtime = new SessionRuntime({
  storage: memoryAdapter,
  userId: "user-123",
  otelExporter: new OtelApiExporter(),
});

The bridge is destination-agnostic: the same OtelApiExporter lands in Honeycomb, Datadog's OTLP intake, Grafana Tempo, and any self-hosted OpenTelemetry Collector — only the OTLP url and auth headers in step 1 differ. For Datadog APM, point OTLP_ENDPOINT at the agent's OTLP intake (http://localhost:4318/v1/traces by default). For Tempo or a self-hosted Collector, point at the collector's OTLP HTTP receiver.

pleach.tenant_id attribute

Every span carries pleach.tenant_id when runtime.tenant is configured, or when an outbound HTTP call rides through the withTenantHeader adapter. The attribute is set at span-start from the active tenant context, so it appears on all four span types without per-call wiring on your side.

This is the load-bearing field for per-tenant trace queries — the Honeycomb / Tempo / Datadog query WHERE pleach.tenant_id = $1 gives you a tenant's full trace history without joining to the audit ledger. See Tenant facet.

Audit gate

audit:otel-noop-soak runs in CI to catch span-emit drift. It asserts two things:

1. The substrate emits exactly one of each of the four span types when a turn runs against an in-process exporter. A refactor that silently stops emitting one (a missing tracer.startSpan after a control-flow change) fails the gate.
2. The noop default — no exporter wired, no SDK started — stays genuinely noop. No allocations from span construction, no buffered records hanging on a process-global.

For consumers, the gate is informational: it tells you the substrate's emit contract is enforced upstream. You don't run it; the upstream CI does, and a release that ships with it red is the release that breaks your trace tree.

What not to do

• Don't add wall-clock timestamps as span attributes. Spans carry their own timing via startTimeUnixNano / endTimeUnixNano; a second clock disagrees with the first about subtle things (NTP skew, monotonic vs wall).
• Don't emit your own per-call span at a custom seam. The llm.invocation span is the cardinality-correct one — one per audit row, one per actual provider call. A custom span at a provider-wrapper site double-counts; a span in a stream observer fires per chunk and floods the trace tree.
• Don't share a single OTLP exporter across worker processes without configuring the SDK for it. The default batch processor isn't fork-safe; each worker needs its own NodeSDK.start() after the fork, or the parent's exporter ends up holding spans the workers emit.
• Don't disable pleach.tenant_id to "reduce cardinality." It's the join key every tenant-scoped trace query reads. Cardinality control belongs at the exporter's sampler, not at the attribute source.

Where to go next