# Playground & devtools (/docs/playground-and-devtools) This page is direct about what's in the box today and what isn't. Some peers (Mastra Studio, Inngest dev server, VoltOps Console, the OpenAI Tracing UI) ship a `localhost:NNNN` playground out of the box. Pleach doesn't — by design. The audit row, the OTel span, and the event log all land in **your** database / **your** OTel backend, and you pick the dashboard. That's a real trade. Here's what each side gets you, what ships today for the in-process / in-browser case, and where to reach when you want a richer dev surface. ## The trade [#the-trade] | | Hosted-dashboard peer | Pleach | | -------------------------------------------------- | -------------------------------------------- | ------------------------------------------------------------------ | | Where data lands | Vendor service ↔ your code | Your Postgres + your OTel backend | | Vendor lock-in | Some (account, retention, billing per-trace) | None — schema is documented, rows are yours | | Dashboard out of the box | ✅ `localhost:NNNN` UI on `npm run dev` | ❌ — bring Grafana / Honeycomb / Tempo / Datadog / your own panel | | Compliance posture (HIPAA / GDPR / data residency) | "Trust the vendor's SOC 2" | Self-hosted by construction; no vendor boundary to add to your DPA | | Per-tenant cost rollup | Per-vendor pricing model | One `GROUP BY tenant_id` against your `auditable_call` table | | Replay-deterministic export | Vendor-shaped | Your DB → your CI → bit-exact `StreamEvent` corpus | The honest summary: **if you want a polished dashboard with no config, peers are better. If you want the data in your hands with no third party, pleach is better.** The two postures don't combine — you pick. ## What ships today [#what-ships-today] Three real surfaces, none of them a hosted dashboard. ### 1. `window.__HARNESS_DEVTOOLS__` — browser console [#1-window__harness_devtools__--browser-console] In development the runtime exposes a debugging interface on `window.__HARNESS_DEVTOOLS__`. Inspect the live session, walk back through checkpoints, force-sync the outbox without leaving the browser console. Gate behind `NODE_ENV !== "production"` so the surface doesn't ship in production bundles. Full surface on [DevTools](/docs/devtools). ### 2. `inspectRuntime()` — typed runtime introspection [#2-inspectruntime--typed-runtime-introspection] Programmatic, not visual. `inspectRuntime(runtime)` returns a typed snapshot of which `contribute*` hooks the plugins on a constructed `SessionRuntime` actually implement — of the \~45 known hooks, which a registered plugin covers, which are unwired, and which use deprecated forms slated for removal. Useful when a plugin "doesn't seem to do anything" and you're not sure if it actually wired. Full surface on [Runtime inspector](/docs/runtime-inspector). ### 3. The event log + OTel spans — your dashboard of choice [#3-the-event-log--otel-spans--your-dashboard-of-choice] Every turn writes to `harness_event_log` (or whatever you configured); every stage emits an OTel span with parent threading. Pipe to: * **Grafana + Tempo + Loki** — self-hosted; the panel template for the four-stage lattice lives at [observability](/docs/observability#grafana-panel-templates). * **Honeycomb / Datadog / Lightstep** — the OTel spans flow as standard `llm.invocation` / `tool.execution` / `session.turn` / `graph.stage`; no Pleach-specific exporter required. * **A Next.js page in your own app** — read from `harness_event_log` directly, render with whatever you like. The `auditable_call_row` schema is documented at [the AuditableCall row](/docs/auditable-call-row). * **A SQL notebook** — Hex, Mode, Hightouch. The grain is per-turn; you join to billing on `tenant_id` + `turn_id`. ## What ISN'T in the box [#what-isnt-in-the-box] Setting expectations explicitly so you don't go looking: * **No `npm run pleach:dev` that opens `localhost:NNNN`.** The closest analog is `npx pleach init`'s scaffolded app — it gives you a working chat at `/`, not a dashboard at a separate port. * **No hosted SaaS dashboard.** `@pleach/observe` ships destination plugs (Postgres, Supabase, OTel, memory) but no Pleach-Hosted destination today. If a Pleach-Hosted offering ships in the future, it will be opt-in and have its own separate SOC 2 boundary; the self-hosted path will remain first-class. * **No "click on a turn, see the stages light up" visualizer.** The OTel spans carry that data; pipe them to a tracing UI that can render the parent-child topology. * **No replay-in-browser UI.** `@pleach/replay`'s `replayTurn(turnId)` is a programmatic surface that returns a bit-exact `StreamEvent` stream. Render it in your own UI or diff it in CI; there's no shipped replay viewer. ## Common dev-loop recipes [#common-dev-loop-recipes] A few patterns people land on: ### See every event during one turn [#see-every-event-during-one-turn] Tail `harness_event_log` filtered by your dev `userId`: ```sql SELECT seq_within_turn, ts, event_type, payload FROM harness_event_log WHERE user_id = 'me@example.com' AND ts > now() - interval '5 minutes' ORDER BY seq_within_turn ASC; ``` ### Walk the checkpoints for a turn [#walk-the-checkpoints-for-a-turn] Programmatic via `runtime.checkpoints.list(sessionId, turnId)`; the [Time travel](/docs/time-travel) page has the full walk. ### See what a plugin actually contributes [#see-what-a-plugin-actually-contributes] `inspectRuntime(runtime).plugins[i].contributions` returns the list of populated `contribute*` hooks. Empty list = the plugin isn't doing anything. ### Confirm a turn is replay-deterministic [#confirm-a-turn-is-replay-deterministic] Run the same turn twice against `HARNESS_MOCK_MODE=true`. The two `StreamEvent` streams should be byte-identical (modulo timestamps). If they're not, you've introduced a non-determinism source — see [Determinism](/docs/determinism) for the audit. ### Test a plugin headlessly, continuously [#test-a-plugin-headlessly-continuously] `npx pleach dev --plugin ./my-plugin.mjs --scenarios ./b.json --watch` drives a battery through your plugin with no browser, folds each turn into a report (the six panels + signals), and re-runs + diffs on every edit. Add `--assert ./expect.json` to gate it in CI. See [`pleach dev` → plugin-test harness](/docs/cli#plugin-test-harness--headless-continuous). ## The `harness` chatinit flag [#the-harness-chatinit-flag] The CLI harness above is one consumer of a surface you can flip on in your **own** runtime. The runtime already exposes the four capture seams (`eventLogWriter`, `onGraphChannelSnapshot`, `auditRowSink`, `onRuntime` → `inspectRuntime`); the `harness` chatinit flag wires them for you and folds each turn into a `TurnReport` — the same six surfaces + derived signals (incl. `capabilities wired n/total`) — handed to `onTurnReport`. Use it for self-tests, CI gates, telemetry, or an in-app `run → observe → edit` agent loop. ```ts import { createPleachRoute } from "@pleach/core/quickstart"; import { evaluateAssertions } from "@pleach/core/harness"; export const POST = createPleachRoute({ // flip on behind your own env flag — off by default in prod harness: process.env.PLEACH_HARNESS === "1" ? { onTurnReport: (report) => { const { passed, failures } = evaluateAssertions( { plugins: report.signals.plugins, scenarios: [report] }, { all: { reachedTerminal: true, notStalled: true } }, ); if (!passed) console.warn("[harness]", failures); }, } : undefined, }); ``` The same flag exists on `new SessionRuntime({ harness: { onTurnReport } })` for the non-route path (event-log + channel-snapshot signals; the route flag additionally folds in the `inspectRuntime` capability signal, since it holds the per-request runtime). Audit rows are folded by the CLI harness, not the chatinit flag (the audit sink is process-global) — so the audit-ledger predicates (`minAuditRows`, `everyAuditRowClassified`, `everyAuditRowTenantScoped`, `exactlyOneTerminalSynthesis`, `singleFamily`) only apply on the CLI path; from the chatinit flag, assert the event-log + channel signals (`reachedTerminal`, `notStalled`, `eventTypes`, `snapshotChannels`, `minCapabilitiesWired`). `onTurnReport` is fail-soft — a throwing callback can never break the turn. `@pleach/core/harness` also exports the pure pieces so you can wire the loop over the existing seams however you like: | Export | Purpose | | ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `foldTurnReport(capture)` | Fold one turn's captured signals into a `TurnReport`. | | `foldTurnReportFromEvents(events)` | Same, from a raw `StreamEvent` / NDJSON stream. | | `createHarnessCapture({ onTurnReport, inspect })` | Composable sinks (`eventLogWriter` / `onGraphChannelSnapshot` / `auditRowSink`) that fold + fire per turn. | | `projectInspector(report)` | The `inspectRuntime` → `{ plugins, wired, total, notContributed }` projection. | | `evaluateAssertions(report, spec)` | `{ passed, failures }` against an expectations spec. | | `diffReports(prev, next)` | Human-readable per-scenario change lines (the `--watch` signal). | | `buildReportMarkdown(report, { verbose })` | Render a battery report as Markdown. `{ verbose: true }` appends the audit-ledger table (`seq · stage · callClass · family · model · transport · status`), the runtime-log stream, and tool cards — all folded from generic `TurnReport` fields. Default output is byte-identical to the non-verbose form. | | `detectEdgeCases(report)` | The answer-quality battery — pure detectors over a `TurnReport` that flag a shipped-but-broken answer: XML/`` leak, delegate-envelope leak, plan-narration-as-answer, thin/empty/duplicated/divergent answer, fabricated citation ref, no-runtime degradation, guard truncation. Returns `EdgeCaseFinding[]`. Three detectors accept hints (`toolInputKeyHints` / `dataSourceHints` / `additionalToolRanMarkers`) and domain checks register via a `supplementaryChecks` slot, so a consumer with its own tool vocabulary reuses the battery without forking it. | | `hasSubstantiveAnswerBody(text)` | The pure predicate the battery and the answer-sufficiency backstop both build on — does a draft carry a real answer body (table / ≥2 headers / citation / ≥2 analytical paragraphs after subtracting `PLAN_NARRATION_PATTERNS`) vs. pure "Let me search…" narration. Zero domain vocabulary. | | `getDiagnosticsBus()` / `DiagnosticsBus` | A `messageId`-keyed in-memory pub/sub bus (`publishCrumb` / `publishTurnReport` / `subscribe` / `endTurn`) that replays a turn's console breadcrumbs + folded `TurnReport` to an out-of-band consumer without touching the chat stream. Redaction is injectable (`redactPatterns`; core default = none). | ## Where to go next [#where-to-go-next]