# Performance (/docs/performance) The runtime's overhead per call is small but not zero. This page documents what the substrate does for you, what it expects you to configure, and where to look when latency or throughput matter. ## Where the time goes [#where-the-time-goes] A single LLM call through the runtime walks five distinct phases. Knowing the rough budget for each helps when you're profiling. | Phase | Typical share | Optimized by | | ---------------------------------------------------------------------- | ------------------------- | --------------------------------------------------------- | | Prompt composition | 1–5 ms | Static-vs-runtime-aware split; pre-resolved core baseline | | [Fingerprint](/docs/fingerprint) compute + [cache](/docs/cache) lookup | `<1 ms` | `sha256` is fast; key fields are pre-canonicalized | | Provider invoke + stream | 200 ms – tens of seconds | Provider-side; runtime is along for the ride | | [Channel](/docs/channels) / reducer updates | `<1 ms` per chunk | Channel kinds picked for low-overhead writes | | [Audit ledger](/docs/audit-ledger) write | `<1 ms` (fire-and-forget) | Batched at the adapter; never blocks the turn | The provider call dominates. Everything else stays in the single-millisecond range. If your per-call overhead is multi-tens of ms, something custom is the cause — usually a slow plugin hook, a heavy runtime-aware prompt contribution, or a stream observer doing work that should be async. Concretely: a turn that takes 4.2s end-to-end against a model that reports 4.1s of provider latency on the ledger row leaves \~100ms of runtime overhead — well within the table's budget (1–5ms prompt + \<1ms fingerprint + \<1ms channels per chunk + \<1ms ledger), with the slack spent on the chunk-by-chunk channel writes during streaming. A turn that takes 4.5s against the same 4.1s provider latency leaves \~400ms unaccounted — that's the signal to profile plugin hooks, because the substrate's own phases don't add up to that number. ## Fingerprint-based dedup [#fingerprint-based-dedup] Every call computes a fingerprint. Two calls with the same fingerprint hash are interchangeable — the runtime can serve the second from the first's recorded result. The cache layer isn't built in; it's a contract that lets you plug one in. The simplest pattern: ```typescript import { computeFingerprint, canonicalize, sha256 } from "@pleach/core/fingerprint"; const fp = computeFingerprint(fingerprintInput); // The Fingerprint carries typed component fields (promptHash, toolsHash, …); // derive the canonical cache key with the same canonical-JSON + sha256 helpers // the fingerprint uses internally, so keys join cleanly across processes. const fpKey = sha256(canonicalize(fp)); const cached = await myCache.get(fpKey); if (cached) return cached; const result = await provider.execute(config); await myCache.set(fpKey, result, { ttl: 3600 }); return result; ``` Wrap this in a provider decorator and you get cache-hit reuse without changing application code. The fingerprint contract guarantees the cache is sound — same hash means same output for the same model. ### What survives a cache hit [#what-survives-a-cache-hit] A cache hit is byte-identical to the recorded result; the audit ledger still writes a row, but the row carries a `cacheHit: true` flag in the payload. Two implications: 1. Latency drops to a single network round trip (cache fetch + audit write). 2. The ledger remains complete — every call is still recorded, even cached ones. `GROUP BY model` rollups stay accurate. ### What invalidates the cache [#what-invalidates-the-cache] The fingerprint includes `pleachVersion`, so substrate upgrades invalidate automatically. Within a version, anything in the fingerprint's IN set (model, messages, tools, system prompt, temperature bucket, seed, family, call class, runtime mode, tenant, active safety policies) is the invalidation surface. A new prompt contribution invalidates the cache; toggling a safety policy invalidates the cache; changing model id invalidates the cache. That's the contract. ## Prompt caching (provider-side) [#prompt-caching-provider-side] Distinct from the fingerprint cache: provider-side prompt caching is what Anthropic and OpenAI offer to discount the input-token bill for repeat prefixes. Two integration paths: ### Native via `AnthropicSdkProvider` [#native-via-anthropicsdkprovider] The Anthropic provider passes through Anthropic's prompt-cache control headers. Mark cacheable sections with `cache_control` markers in the system prompt; the provider forwards them. ```typescript const runtime = new SessionRuntime({ provider: new AnthropicSdkProvider({ apiKey: process.env.ANTHROPIC_API_KEY!, model: "claude-sonnet-4-5", // Prompt cache is on by default for cacheable content blocks. }), // ... }); ``` ### Via the AI SDK [#via-the-ai-sdk] The AI SDK's `experimental_providerOptions` field on `streamText` threads through `AiSdkProvider.providerConfig`. Use it to pass prompt-cache flags per call. In both cases, prompt caching and fingerprint dedup compose: the fingerprint cache replays the whole response; prompt caching discounts the prefix tokens on a fresh provider call. Use both. ## Batching tool calls [#batching-tool-calls] `ToolBatchExecutor` groups concurrent tool calls per the batching strategy declared on each tool. The default inferred strategy is conservative (serial); override on tools that are safe to parallelize. ```typescript // lib/tools/fetchDocument.ts import { defineTool, Semaphore } from "@pleach/core"; export const fetchDocument = defineTool({ name: "fetch_document", description: "Fetch one document by id.", inputSchema: z.object({ id: z.string() }), async execute(input, ctx) { /* ... */ }, // @ts-expect-error — extended field batching: { strategy: "parallel", maxConcurrency: 5 }, }); ``` Choosing strategy: | Strategy | When | | ---------- | ------------------------------------------------------------------ | | `serial` | Tool mutates external state, or order matters | | `parallel` | Read-only tools that can fan out — searches, fetches | | `batched` | Tool's underlying API accepts arrays — issue one call for N inputs | Use `Semaphore` inside `execute` for hand-rolled concurrency limits when the tool itself fans out further. A search tool that fetches the top hits in parallel, capped at four in flight at once: ```typescript // lib/tools/searchCorpus.ts import { defineTool, Semaphore } from "@pleach/core"; const gate = new Semaphore(4); export const searchCorpus = defineTool({ name: "search_corpus", description: "Search the knowledge base and hydrate the top hits.", inputSchema: z.object({ query: z.string(), topK: z.number().default(8) }), async execute({ query, topK }, ctx) { const hits = await index.search(query, { limit: topK, signal: ctx.signal }); return Promise.all( hits.map(async (h) => { await gate.acquire(); try { return await fetchDocument(h.id, ctx.signal); } finally { gate.release(); } }), ); }, }); ``` The semaphore lives at module scope, so it caps concurrency across every concurrent invocation of `search_corpus` in the process — not just within one call. ### Fan-out, fan-in subagents [#fan-out-fan-in-subagents] When a turn needs three independent sub-analyses, spawn them in parallel and await the joined result. The runtime tracks each [subagent](/docs/subagents) under its own `turnId`; the audit ledger keeps the attribution clean: ```typescript async execute(input, ctx) { const [summary, facts, label] = await Promise.all([ ctx.spawnSubagent({ tool: "summarize", input: { docId: "doc-abc123" } }), ctx.spawnSubagent({ tool: "extract_facts", input: { docId: "doc-abc123" } }), ctx.spawnSubagent({ tool: "classify", input: { docId: "doc-abc123" } }), ]); return { summary, facts, label }; } ``` Three rows in `harness_auditable_calls`, three `subagent.completed` events, one parent turn — wall-clock latency is the slowest branch, not the sum. ## Lazy module loading [#lazy-module-loading] Hosts mid-migration use `setHarnessModuleLoader` with dynamic imports. The runtime caches resolved modules — each key resolves once per runtime lifetime, then stays in memory. For cold-start-sensitive deployments (Fluid Compute, Lambda, Cloudflare Workers), the first call after cold boot pays the dynamic-import cost for every key it hits. Two mitigations: 1. **Warm critical paths in your handler init.** Resolve the keys the first turn will need before the request lands. 2. **Use the typed config surface where possible.** `provider`, `tools`, `plugins`, `metaToolNames`, and the storage adapters never go through dynamic import — they're direct references. ## Channel write overhead [#channel-write-overhead] Channel writes are O(1) for `LastValue` / `EphemeralValue` / `NamedBarrier`, O(N) for `Topic` / `BinaryOperatorAggregate` (N = size of the existing accumulator), and O(log N) for `DataChannel` (LRU map operations). For high-throughput per-step writes, prefer `Topic` over `BinaryOperatorAggregate` — appending a single item is O(1) amortized, whereas a `BinaryOperatorAggregate` write applies the reducer over the existing value. A concrete case: a `search_corpus` result with 500 hits, fan-in through `BinaryOperatorAggregate + appendReducer`, applies the reducer once per write — the 500th write copies a 499-element array. The same result through a `Topic` lands each item in O(1). For a single 500-hit turn the difference is irrelevant; for a 5000-hit turn it's the difference between a turn that lands in 100ms and one that lands in 2s of pure channel-write overhead. When the reducer dominates (e.g. a per-message dedup pass over many thousand messages), bucket the writes — accumulate in an ephemeral channel within a step, flush once at end-of-step. ## Stream observer cost [#stream-observer-cost] Observers fire on every chunk. `onChunk` is sync-only by contract, so the cost is bounded; the runtime won't await an observer that returns a promise. For observers that need to do something expensive (network call, DB write), pattern: observe synchronously, emit a named-channel envelope, handle the expensive work in a post-turn node. ```typescript contributeStreamObservers: () => [{ name: "detect-citation", onChunk(chunk, ctx) { if (looksLikeCitation(chunk.text)) { ctx.emit("citations.detected", { fragment: chunk.text }); } return { verdict: "continue" }; }, }], ``` The post-turn node reads `citations.detected` and does the network resolution — async work happens off the hot path. ## Audit ledger batching [#audit-ledger-batching] The `ProviderDecisionLedger` contract permits batching: up to 50 ms or 32 records, flushed unconditionally at end-of-turn (writing one [auditable call row](/docs/auditable-call-row) per record). Reference Supabase adapter implementations batch by default. For very high-throughput deployments (multi-thousand-RPS), the adapter is the natural place to add a queue + worker pattern — the contract is fire-and-forget, so a queued write that lands later still satisfies the invariants. ## DevTools-driven profiling [#devtools-driven-profiling] When a turn is mysteriously slow in development: ```javascript __HARNESS_DEVTOOLS__.events({ types: ["step.start", "step.end"] }); // Walk pairs to see per-step latency. __HARNESS_DEVTOOLS__.ledger().map((r) => ({ call: r.callClass, model: r.modelId, latency: r.latencyMs, })); // Per-call latency breakdown. ``` The per-call latency on the audit row is provider-side latency. Subtract from observed end-to-end and you get a runtime overhead estimate. Typical overhead is single-millisecond; multi-tens of ms is the signal to look for a hot-path plugin. The DevTools ledger view also surfaces `cacheHit` — a turn with three rows all showing `cacheHit: true` should report effectively zero provider latency (the fingerprint replay path doesn't dial out), so a cache-hit row reporting 500ms of `latencyMs` is itself a bug signal (likely a stale recording that's slower than the live provider). ## When to bring in `@pleach/gateway` [#when-to-bring-in-pleachgateway] The gateway SKU is the right pick when: * Per-tenant routing rules need to be policy-driven, not hardcoded. * Cost attribution needs to roll up across providers (Anthropic rate-limited fallback to OpenAI, for example). * Observability needs to feed an external sink (Datadog, Honeycomb) at call granularity. It's a plugin against `@pleach/core` — no architectural change, just an extra contribution in your plugin set. ## Where to go next [#where-to-go-next]