# agentInstrumentation (/docs/recipes/agent-instrumentation)



This is the do-it-yourself instrumentation path. You hold a
`SessionRuntime` and you want to watch how each turn runs —
stage transitions, recovery dispatches, retries, per-LLM-call
cost — without adopting a recipe wrapper or threading a recorder
argument through your code.

Two surfaces carry the signal. The durable `runtime.events` bus
carries lifecycle and per-call events for cross-cutting
subscribers. The per-turn `StreamEvent` iterator carries the
same lifecycle kinds inline with the turn, for UIs that render
from the stream. Pick the one that matches where your subscriber
lives.

Best fit: **any `@pleach/core` consumer** who wants
metrics or audit rows from an existing runtime without rewriting
the turn loop. For the wrapped-chatbot version of this story, see
[`observableChatbot`](/docs/recipes/observable-chatbot). For the
no-`@pleach/core` brownfield version, see
[BYOK observability](/docs/recipes/byok-observability).

## Subscribe to the runtime bus [#subscribe-to-the-runtime-bus]

`runtime.events.on(kind, handler)` registers a handler for one
event kind on the durable bus. It returns an unsubscribe
function. The bus is the right surface for a metrics exporter, a
cost meter, or an alerting hook — anything that outlives a single
turn.

```typescript
const offModel = runtime.events.on("model.called", (e) => {
  // { provider, model, callClass, inputTokens, outputTokens, costUSD, latencyMs }
  meter.record(e.model, e.latencyMs);
});

const offError = runtime.events.on("error", (e) => {
  // { error, context }
  alert(e.error);
});

// later, on teardown:
offModel();
offError();
```

`model.called` fires once per LLM call. As of the seam-path
landing it fires for **both** call sites in a turn:

* the main agentic turn (the converse / synthesize loop), and
* each seam call — the dedicated `synthesize`, `reasoning`, and
  `utility` seam invocations the graph makes around the main
  turn.

That distinction matters for cost. See
[What fires today](#what-fires-today) below before you sum
`costUSD` across rows.

## Subscribe inline with the turn [#subscribe-inline-with-the-turn]

When your subscriber already owns the turn loop — a React UI, an
SSE forwarder — read the same lifecycle kinds off the
`StreamEvent` iterator instead. `@pleach/react`'s `useChat`
exposes them through `onEvent`:

```typescript
useChat({
  onEvent: (e) => {
    if (e.type === "stage.transition") {
      // { from: StageId | null, to: StageId }
      track("stage", e.to);
    }
    if (e.type === "turn.completed") {
      // { outcome: "ok" | "error" | "interrupted", durationMs }
      track("turn", e.outcome, e.durationMs);
    }
  },
});
```

The same kinds arrive on the raw iterator if you do not use
`useChat`:

```typescript
for await (const e of runtime.executeMessage(sessionId, prompt)) {
  if (e.type === "recovery.fired") {
    // { arm: RecoveryDispatchArm }
    track("recovery", e.arm);
  }
}
```

These are `StreamEvent` members, not `runtime.events` bus events.
The full catalog — payload shapes and when each fires — is on
[Stream events → Execution lifecycle](/docs/stream-events#execution-lifecycle).
`model.called` is the exception: it lives on the durable bus, not
the stream, because cost attribution is a long-lived concern.

## Bridge model.called to a destination [#bridge-modelcalled-to-a-destination]

`observeSink` turns the `runtime.events.on("model.called", ...)`
subscription into audit rows with one line. It is a handler
factory: `observeSink({ destinations })` returns a `model.called`
handler that maps each event to an `ObserveRow` and writes it to
every destination you pass.

```typescript
import { observeSink } from "@pleach/observe";
import { memory, postgres } from "@pleach/observe/destinations";

const store = memory();

runtime.events.on(
  "model.called",
  observeSink({
    destinations: [store, postgres({ pgClient, tableName: "pleach_observe_calls" })],
  }),
);

// every LLM call now writes one ObserveRow to both destinations
```

The mapping is `runtimeEventToObserveRow` — a pure function from
a `model.called` event to an `ObserveRow`. `observeSink` calls it
for you; import it directly if you want the row without the write
(to enrich `tags` before handing it to your own sink, say):

```typescript
import { runtimeEventToObserveRow } from "@pleach/observe";

runtime.events.on("model.called", (e) => {
  const row = runtimeEventToObserveRow(e);
  myKafkaTopic.publish({ ...row, tags: { region: "us-east-1" } });
});
```

`ObserveRow` is the same shape the brownfield `recordCall` path
writes — a strict subset of `@pleach/core`'s `AuditableCall` v13
record. See [`@pleach/observe`](/docs/observe) for the row fields,
the four destinations, redaction, and sampling.

## What fires today [#what-fires-today]

`model.called` fires for both the main turn and the seam calls,
but the two carry cost differently.

| Source                                                    | When it fires | Tokens + latency | costUSD                                                           |
| --------------------------------------------------------- | ------------- | ---------------- | ----------------------------------------------------------------- |
| Seam calls (`synthesize` / `reasoning` / `utility` seams) | Every turn    | Real             | Real — computed from the family's per-1M-token rates              |
| Main agentic turn (converse / synthesize loop)            | Every turn    | Real             | `0` sentinel — derive downstream from the `llm.turn` token counts |

The seam-call rows are the ones with real `costUSD`. The
main-turn row reports tokens and latency but carries `0` for
`costUSD` by design — cost for the main turn is derived
downstream from `llm.turn` token metadata, not stamped on the
`model.called` event. If you sum `costUSD` naively across all
`model.called` rows in a turn, the main-turn row contributes
nothing; that is expected, not a dropped row.

The lifecycle stream events fire every turn regardless of the
above:

* `turn.started` / `turn.completed`
* `stage.transition` (and `stage.lattice_violation` when a
  transition leaves the advertised lattice)
* `recovery.fired` — `arm` ∈ `"zeroToolRecovery"` |
  `"allToolsFailedMissingParams"` | `"maxStepsHit"`
* `retry.attempted` — `{ reason, attempt }`
* `stream.first_chunk` / `stream.completed` — TTFB and total
  stream duration

So a UI can reconstruct the per-turn lattice walk and report TTFB
and total latency from the stream alone, with no separate metrics
pipeline.

## Common gotchas [#common-gotchas]

* **`model.called` is a bus event, not a stream event.**
  Subscribe with `runtime.events.on("model.called", ...)`. It does
  not arrive in the `executeMessage` iterator or in
  `useChat({ onEvent })` — those carry the lifecycle stream kinds
  (`stage.*`, `turn.*`, `recovery.fired`, `retry.attempted`,
  `stream.*`), not the per-call cost signal.
* **The main-turn `costUSD` is a `0` sentinel.** Real cost on the
  main turn is derived from `llm.turn` token counts downstream;
  the seam-call rows are the ones carrying computed `costUSD`.
  Don't treat a `0`-cost main-turn row as a bug.
* **`observeSink` needs no `init`.** Unlike the brownfield
  `recordCall` path, `observeSink({ destinations })` writes
  straight to the destinations you pass — there is no
  process-singleton to initialize first.
* **`runtime.events.on` returns an unsubscribe function.** Call it
  on teardown for long-lived processes; a leaked handler keeps the
  destination write alive past the runtime you meant to retire.

## See also [#see-also]

* [Stream events](/docs/stream-events) — the full `StreamEvent`
  catalog, including the execution-lifecycle kinds this page
  subscribes to.
* [`@pleach/observe`](/docs/observe) — `observeSink`,
  `runtimeEventToObserveRow`, the `ObserveRow` shape, and the four
  destinations.
* [`observableChatbot`](/docs/recipes/observable-chatbot) — the
  same observability story as a chatbot recipe wrapper.
* [BYOK observability](/docs/recipes/byok-observability) — the
  brownfield path for loops not yet on `@pleach/core`.
* [Observability](/docs/observability) — the runtime-side
  observability surface.