# Troubleshooting (/docs/troubleshooting) A failure-mode reference. Each entry pairs an observed symptom with the most likely cause and the concrete fix. When a symptom has multiple causes, they're ordered by frequency. If a failure isn't here, the [Error codes](/docs/error-codes) catalog covers the structured 1xxx–9xxx ranges with recovery hints. CI-time `audit:*` gate failures are covered by [Audit gates](/docs/audit-gates). ## Runtime construction [#runtime-construction] ### `[UXParity:metaToolNames-config-missing]` at startup [#uxparitymetatoolnames-config-missing-at-startup] **Cause.** `SessionRuntimeConfig.metaToolNames` not passed; the runtime falls back to an empty set and continuation / fabrication guards depending on the set silently disable. **Fix.** Pass the set explicitly: ```typescript const runtime = new SessionRuntime({ metaToolNames: new Set(["set_step_complete", "wait_for_jobs"]), // ... }); ``` See [SessionRuntime](/docs/session-runtime). ### `HarnessModuleLoaderUnregisteredError` [#harnessmoduleloaderunregisterederror] **Cause.** `executeMessage` was called before `setHarnessModuleLoader(...)` registered a loader; happens in hosts mid-migration from legacy orchestrator integration. **Fix.** Register the loader at startup. The error message names the first key it tried to resolve — implement that arm at minimum. See [Host adapter](/docs/host-adapter). ### `UnknownSafetyPolicyError` [#unknownsafetypolicyerror] **Cause.** A policy id in `enabledSafetyPolicies` doesn't match any policy registered through `contributeSafetyPolicies()`. **Fix.** Check the spelling against the plugin that ships the policy. Programmer-error class — surfaces at construction, not on the first turn. ### Bisecting "is the bug in the runtime or my tool?" [#bisecting-is-the-bug-in-the-runtime-or-my-tool] Set `HARNESS_MOCK_MODE=true` and re-run the failing input. Mock mode replaces the provider with a deterministic stub but leaves your tools, plugins, and runtime config untouched: ```bash HARNESS_MOCK_MODE=true npm run dev # In another shell: curl -X POST http://localhost:3000/api/chat \ -H 'content-type: application/json' \ -d '{"sessionId":"session-018f-7a","message":"fetch doc-abc123"}' ``` If the bug reproduces under mock mode, it's in tool code or plugin code (the provider isn't running). If it disappears, the bug is in provider config or in the cascade. Never ship code with `HARNESS_MOCK_MODE=true` set — it's a debug-only switch. ### Runtime works in dev, throws in production on first request [#runtime-works-in-dev-throws-in-production-on-first-request] **Cause.** `HARNESS_MOCK_MODE=true` was set in dev but not in production; the runtime tried to construct a real provider / storage adapter with missing env vars. **Fix.** Verify production env. The mock-mode env var should never be set in production; the real `OPENROUTER_API_KEY` (or `ANTHROPIC_API_KEY` / `OPENAI_API_KEY`) plus `SUPABASE_URL` / `SUPABASE_SERVICE_ROLE_KEY` should be. ### OTel exports are empty, but the runtime is firing turns [#otel-exports-are-empty-but-the-runtime-is-firing-turns] **Cause.** The OTel SDK was started after `new SessionRuntime(...)` was constructed. The runtime reads the global tracer provider at construction time. **Fix.** Call `sdk.start()` before constructing the runtime. If the runtime is built in a module-level constant, hoist the SDK start to a top-level side-effect import before any runtime import. See [OTel observability](/docs/otel-observability). ## Streaming [#streaming] ### Stream silently disconnects after \~30 s in production [#stream-silently-disconnects-after-30-s-in-production] **Cause.** SSE response is being buffered by a reverse proxy or CDN. Most common with Vercel's default response handling when `Cache-Control` isn't set. The diagnostic: open the network panel and watch the response — if `data:` lines arrive in a single burst at \~30s instead of trickling, the proxy is buffering; if no bytes arrive at all before the timeout, the connection is being killed upstream. Buffered streams break the chunked rendering; the fix below addresses both. **Fix.** Set headers on the SSE response: ```typescript return new Response(stream, { headers: { "Content-Type": "text/event-stream", "Cache-Control": "no-cache, no-transform", "Connection": "keep-alive", "X-Accel-Buffering": "no", // nginx }, }); ``` ### `tool.started` fires but `tool.completed` never does [#toolstarted-fires-but-toolcompleted-never-does] **Cause.** Tool's `execute` is awaiting an external resource that never resolves (no timeout, no abort handling). The diagnostic step: pull the `toolCallId` from the `tool.started` payload and query the audit ledger for `payload->>'toolCallId' = $1` — if no row exists, `execute` never returned at all (and the provider call was never made); if a row exists with `outcome.status === "failed"`, the tool errored but the failure event was lost (likely durable flush wasn't registered). **Fix.** Honor `ToolContext.signal` in every fetch / spawn: ```typescript async execute(input, ctx) { const res = await fetch(url, { signal: ctx.signal }); // ... } ``` The user pressing stop cancels the AbortController; tools that ignore the signal keep running and the stream looks frozen. ### `message.delta` chunks arrive, then a `content.correction` replaces them [#messagedelta-chunks-arrive-then-a-contentcorrection-replaces-them] **Cause.** Not a bug. A post-stream fabrication guard detected phantom tool references or another correctable issue and rewrote the streamed content. **Fix.** Treat `correctedContent` as the authoritative final text — replace the last assistant message wholesale. See [Stream events](/docs/stream-events). ### Stream stops mid-token with `stream.truncated` [#stream-stops-mid-token-with-streamtruncated] **Cause.** A degeneration guard (entropy collapse, phrase loop, substring repetition) detected runaway output and stopped the stream. **Fix.** Surface the truncation reason to the user (`entropy_collapse` typically means the model lost coherence; `phrase_loop` means it got stuck repeating). The `partialLength` field tells you how much was emitted before truncation. ## Tools [#tools] ### `Tool not found` (code `1001`) [#tool-not-found-code-1001] **Cause.** Tool name in `createSession({ tools: { enabled: [...] } })` doesn't match any registered tool, or registration ran after the session was created. **Fix.** Register tools before constructing the runtime (or before any `createSession` call). Verify with DevTools: ```javascript __HARNESS_DEVTOOLS__.tools().map((t) => t.name); ``` ### Tool fires the wrong arguments [#tool-fires-the-wrong-arguments] **Cause.** The model is producing arg shapes that don't match your Zod schema; the validator throws and `tool.failed` fires with the validation error. **Fix.** Two paths. 1. **Tighten the description.** The model uses `description` to pick args; ambiguous descriptions produce ambiguous args. 2. **Loosen the schema.** Use `z.string().or(z.number())` for fields where the model produces both, then normalize in `execute`. ### Tool runs serially even though it's safe to parallelize [#tool-runs-serially-even-though-its-safe-to-parallelize] **Cause.** Default batching strategy is conservative; tools inherit `serial` unless overridden. **Fix.** Mark explicitly: ```typescript defineTool({ name: "fetch_paper", // @ts-expect-error — extended field batching: { strategy: "parallel", maxConcurrency: 5 }, // ... }); ``` See [Tools](/docs/tools) for the strategies. ### Outbound HTTP from a tool doesn't carry the tenant header in a tenant-scoped runtime [#outbound-http-from-a-tool-doesnt-carry-the-tenant-header-in-a-tenant-scoped-runtime] **Cause.** The fetch client wasn't wrapped with `withTenantHeader`. The runtime stamps `tenant_id` on internal write sites; outbound HTTP is consumer-owned by default. **Fix.** Wrap the fetch client at construction (`withTenantHeader(fetch, runtime.tenant)`) and pass the wrapped client to whatever performs the outbound call — the tool's `fetch`, the SDK client, the gateway adapter. See [Tenant facet](/docs/tenant-facet). ## Storage and sync [#storage-and-sync] ### `Version conflict` (code `3001`) on every write after a network blip [#version-conflict-code-3001-on-every-write-after-a-network-blip] **Cause.** The client's version vector has drifted behind the server's; the conflict-detection layer is doing its job. **Fix.** Pull the remote vector, merge, and re-push. The `SyncCoordinator` does this automatically; the symptom usually means a custom adapter is bypassing the coordinator. ### `Session not found` (code `2001`) right after `createSession` returns [#session-not-found-code-2001-right-after-createsession-returns] **Cause.** The runtime's `userId` doesn't match the RLS policy's expected scope. Common when constructing with an anon client and no auth context. **Fix.** Either pass an authed session token (anon client path) or use a service-role client (server path). Verify with a direct DB query — the row exists; RLS is hiding it. ### Sync conflicts surfacing constantly in multi-device setup [#sync-conflicts-surfacing-constantly-in-multi-device-setup] **Cause.** Two clients writing to the same session with overlapping fields and no merger configured. **Fix.** Configure a custom merger in `SyncCoordinatorConfig` that picks `remote` for fields where the server is authoritative (timestamps, server-derived metadata). See [Sync](/docs/sync). ### Outbox grows unbounded; never drains [#outbox-grows-unbounded-never-drains] **Cause.** Network is failing silently; the durable-flush retry budget is exhausted; or `ConnectivityMonitor` is reporting offline when the network is actually up. The diagnostic: check the most recent parked entry's `lastError` — a `3001` code means the network refused every retry (look at the probe URL); a `3003` means the server is rejecting on version mismatch (look at `SyncCoordinator`'s merger configuration); a `3004` means the outbox cap is too small for the write rate. **Fix.** Check `__HARNESS_DEVTOOLS__.syncStatus()` for the `errors` array. Forced sync surfaces the underlying error: ```javascript await __HARNESS_DEVTOOLS__.forceSync(); __HARNESS_DEVTOOLS__.syncStatus().errors; ``` ### A field that used to appear in event log payloads is now blank [#a-field-that-used-to-appear-in-event-log-payloads-is-now-blank] **Cause.** A scrubber matched the field and redacted it before persistence. Scrubbers run at write time; nothing flows past them. **Fix.** List the active scrubbers via the plugin contract surface to find which one matched. If the redaction is overly aggressive, override the matcher pattern — for `KeyedRegex`, narrow the regex; for the bundled scrubbers, register a more specific scrubber that runs first and tags the field as exempt. See [Scrubbers](/docs/scrubbers). ### A custom `GraphProjection` returns stale data on a fresh deployment [#a-custom-graphprojectiont-returns-stale-data-on-a-fresh-deployment] **Cause.** The projection is folding against pre-stamping rows (rows written before the projection's substrate landed) that don't carry the event types the reducer expects. **Fix.** Scope the projection's read with `since: cutoverId` where `cutoverId` is the first row written after the substrate upgrade. For shipped projections in soak, depend on `GraphProjection` directly and treat the bundled implementations as references — their signatures may move between releases. See [Event log projections](/docs/event-log-projections). ## React [#react] ### `useHarness` returns stale messages [#useharness-returns-stale-messages] **Cause.** `HarnessProvider` re-mounted because the `runtime` prop was re-created on every render. **Fix.** Wrap runtime construction in `useMemo`: ```tsx const runtime = useMemo( () => new SessionRuntime({/* ... */}), [userId], // recreate only when scope changes ); ``` ### `useHarnessDevTools` doesn't expose `window.__HARNESS_DEVTOOLS__` [#useharnessdevtools-doesnt-expose-window__harness_devtools__] **Cause.** The hook ran before `HarnessProvider` mounted, or the hook is inside a conditional that didn't fire. **Fix.** Call `useHarnessDevTools()` inside a component that's a descendant of `HarnessProvider`. The hook attaches via the runtime context. ### Chat resets on tenant switch [#chat-resets-on-tenant-switch] **Cause.** `useMemo` dep array missing the tenant id; runtime gets reused across tenants. **Fix.** Add the tenant to the deps: ```tsx const runtime = useMemo( () => new SessionRuntime({/* ... */}), [tenantId, userId], ); ``` This is also the right pattern for tenant isolation — see [Multi-tenant](/docs/multi-tenant). ## Provider [#provider] ### `LLM unavailable` (code `5001`) on the first call [#llm-unavailable-code-5001-on-the-first-call] **Cause.** Provider credentials missing or invalid. The runtime's cascade walked every in-family rung and all failed. **Fix.** Check the env var set per provider — `OPENROUTER_API_KEY`, `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`. Run a smoke test against the provider SDK directly to verify credentials before involving the runtime. ### `Rate limited` (code `5003`) under normal load [#rate-limited-code-5003-under-normal-load] **Cause.** Provider-side rate limit hit on a single API key across all sessions; the cascade walks rungs but every rung shares the same upstream limit. **Fix.** Two paths. 1. **Multi-key routing.** `@pleach/gateway@0.1.0` ships the Phase A surface (`GatewayClient.route`, BYOK fingerprint, family-locked failover for `anthropic`). Other transports are deferred to Phase B per [Gateway](/docs/gateway). For broader coverage today, wire your own router behind the provider adapter or use a third-party gateway (Portkey, LiteLLM proxy, OpenRouter). 2. **Increase the budget.** Contact provider; raise the limit. ### `Token limit exceeded` (code `5005`) on long sessions [#token-limit-exceeded-code-5005-on-long-sessions] **Cause.** Context exceeded the model's window; the runtime emitted a `context.summarized` event and retried, but the retry also exceeded. **Fix.** The runtime's context compaction is automatic but bounded. For sessions that hit it repeatedly, configure a more aggressive compaction strategy or split the session. The diagnostic step: query the ledger for the affected `turnId` and read `payload.tokenUsage.in` — a value approaching the model's window cap (200k for Claude Sonnet 4.5, for example) confirms the prompt itself is the bottleneck rather than the response; a value well under cap with the error firing means a tool result inflated the context mid-turn, and the fix is to truncate the tool result upstream rather than tune compaction. ### Confirming a fallback fired (without trusting logs) [#confirming-a-fallback-fired-without-trusting-logs] The audit row is the ground truth — logs may be rate-limited, the ledger is not. Query for the suspect turn: ```sql SELECT created_at, payload->>'modelId' AS model, payload->>'family' AS family, payload->>'fallbackReason' AS fallback_reason, outcome->>'status' AS outcome FROM harness_auditable_calls WHERE payload->>'turnId' = 'turn-018f-7a-3' ORDER BY created_at; ``` A row with non-null `fallback_reason` confirms the cascade advanced; multiple rows for one `turnId` show the full walk through the family. ### `family-exhausted` event after a fallback cascade [#family-exhausted-event-after-a-fallback-cascade] **Cause.** Every rung in the locked family failed. The runtime emits this rather than silently switching families — see [Architecture](/docs/architecture) §4. **Fix.** Surface a "pick a different model family" prompt to the user. Silent cross-family widening is structurally disallowed by design. ## Audit ledger [#audit-ledger] ### `cacheHit: false` on every call, even ones that should be deduped [#cachehit-false-on-every-call-even-ones-that-should-be-deduped] **Cause.** No fingerprint cache wired. The substrate ships the fingerprint but not the cache layer — that's a contract you implement. **Fix.** Wrap your provider with a fingerprint-keyed cache decorator. See [Performance](/docs/performance). ### Cache hit rate is unexpectedly low despite identical-looking prompts [#cache-hit-rate-is-unexpectedly-low-despite-identical-looking-prompts] **Cause.** One of the four fingerprint gaps differs — `systemPrompt`, `temperature`, `runtimeMode`, or `tenantId`. The cache key is structurally distinct across any gap; identical content with different gap values misses by design. **Fix.** Print the fingerprint via `cacheBackend.metricsSnapshot()` and compare the gap values across the two calls you expected to share a cache row. If the gap difference is structural (cross-tenant pollution, mode change at turn start), that's correctness — the cache is doing its job. If it's accidental (a timestamp leaking into the system prompt), normalize the input via `prepareCacheInputs`. See [Cache](/docs/cache). ### `audit:c8-event-type-allowlist-coverage` fails CI after adding a new event type [#auditc8-event-type-allowlist-coverage-fails-ci-after-adding-a-new-event-type] **Cause.** The new event type doesn't have a matching `Scrubber` allowlist entry. Every persisted event type needs scrubber coverage, even if it's a pass-through. **Fix.** Register a `Scrubber` that allowlists the new event type (a pass-through scrubber is fine if no fields need redaction) via `contributeScrubbers` in the plugin that owns the event type. See [Scrubbers](/docs/scrubbers), [Plugin contract](/docs/plugin-contract). ### Hash-chain verification SQL returns the same row repeatedly [#hash-chain-verification-sql-returns-the-same-row-repeatedly] **Cause.** The recursive CTE's join condition is matching multiple successor rows when no `prev_hash` constraint is present — typically because the chain has a NULL gap mid-stream. **Fix.** Filter to `row_hash IS NOT NULL` on both sides of the join and walk in `record_id` order. A gap of NULL rows is back-compat-expected; the verification resumes at the next stamped row. See [Hash chain](/docs/hash-chain). ### After adding `@pleach/compliance`, CI fails on `audit:tenant-scoping` [#after-adding-pleachcompliance-ci-fails-on-audittenant-scoping] **Cause.** The package brings two CI gates with it. Adopting the package without first wiring `runtime.tenant` (so every write site has a tenant) makes the gate fail. **Fix.** Configure `tenantId` (and optionally `subTenantId`) on `SessionRuntimeConfig` before enabling the compliance plugin. Run `audit:tenant-scoping` locally first; it lists the write sites that need backfill. See [Compliance](/docs/compliance), [Tenant facet](/docs/tenant-facet). ### Ledger rows missing after a crash [#ledger-rows-missing-after-a-crash] **Cause.** Durable flush wasn't registered with `waitUntil` on a serverless function; teardown lost the queued writes. **Fix.** Register the durable-flush pipeline at handler entry: ```typescript import { setWaitUntilImpl } from "@pleach/core/eventLog"; setWaitUntilImpl(ctx.waitUntil.bind(ctx)); ``` See [Event log](/docs/event-log). ### `Two synthesize rows for one turn` invariant violation [#two-synthesize-rows-for-one-turn-invariant-violation] **Cause.** Code somewhere is bypassing `SynthesizeSeamHolder`. Typically a custom plugin reaching across the seam boundary, or a stream observer emitting a synthesized payload. **Fix.** Run `npm run lint:harness-boundary` and `npm run audit:graph-stages` — both will flag the violation. The seam invariants are CI-enforced; if both pass, the violation is in code outside the package's lint scope. ## Schema migrations [#schema-migrations] ### `Schema mismatch` (code `2006`) reading rows in production [#schema-mismatch-code-2006-reading-rows-in-production] **Cause.** The package's schema bundle has advanced past what's applied to the database. **Fix.** Re-run `npx pleach init --apply --target ./supabase/migrations` and apply the new files. The bundle is additive — old files won't change. ### `harness_set_updated_at()` function already exists [#harness_set_updated_at-function-already-exists] **Cause.** A previous migration applied the trigger function, and the bundle's `CREATE OR REPLACE` is being rejected by RLS or permission policies. **Fix.** Run the bundle as the database owner (or service role). The function is shared across tables; redefining it during a migration is intended. ## Where to go next [#where-to-go-next]