# Eval and replay (/docs/eval-and-replay) This page is the DIY pattern against the substrate. For the published SKU references, see [`@pleach/eval`](/docs/eval) and [`@pleach/replay`](/docs/replay). The substrate ships the primitives `@pleach/eval` and `@pleach/replay` build on: deterministic fingerprints, the append-only audit ledger, per-channel checkpoints, and the three `runtimeMode` modes. `@pleach/replay@0.1.0` and `@pleach/eval@0.1.0` are shipping today — `createReplayRuntime`'s four entry points (`replayTurn`, `fromSnapshot`, `fork`, `aggregateMultiTenant`) plus `verifyChainIntegrity` all have real bodies, and the event-granular `ReplayHandle.step()` / `seek()` / `replayTurn()` stepper is wired against `runtime.events.iterate`. This page documents the underlying workflow so you can DIY against the substrate before adopting the SKUs, or read what the SKUs do. This page walks the eval / replay workflow end-to-end. Each section is implementable with only the substrate; sibling-SKU adoption layers typed contracts and scoring on top. ## The three workflows [#the-three-workflows] | Workflow | What you measure | Primitive used | | ------------------- | -------------------------------------------------------- | --------------------------------------- | | **Replay** | Did the same input produce the same output? | Fingerprint + recorded ledger | | **Regression eval** | Does the new version produce the same output as the old? | Fingerprint diff across `pleachVersion` | | **Behavioral eval** | Did the new version produce a *better* output? | Custom scorer + ledger metadata | Replay and regression eval are *correctness* signals — yes/no. Behavioral eval is a *quality* signal — score-based. ## `runtimeMode` is the eval seam [#runtimemode-is-the-eval-seam] The runtime distinguishes three operating modes via `runtimeMode` in the fingerprint key. Picking the right mode is how you opt into the eval workflow. | Mode | Provider calls | Cache reads | Cache writes | | ---------------- | ------------------------------------ | ----------- | ------------ | | `production` | Real | Enabled | Enabled | | `replay` | Intercepted; cached results returned | Enabled | Disabled | | `eval-noncached` | Real | Disabled | Disabled | Set via runtime config or `HARNESS_RUNTIME_MODE`. The mode is in the fingerprint, so a turn recorded in one mode never collides with the cache of another. ## Recording a turn for replay [#recording-a-turn-for-replay] In production mode, every turn writes a full audit-ledger row per LLM call. The row carries the fingerprint, the metadata, the model id, the token usage, and the decision payload — enough to reconstruct the call. ```typescript const runtime = new SessionRuntime({ storage: supabaseAdapter, userId: "user_123", // runtimeMode defaults to "production" }); for await (const event of runtime.executeMessage(sessionId, prompt)) { // ... handle the stream } // After the turn, every call's row is in `harness_auditable_calls`. ``` No additional capture step. The ledger is the recording. ## Replaying a turn [#replaying-a-turn] Build a fresh runtime around the same storage and ledger, open a `ReplayHandle` over the chat's event log via `@pleach/replay`, then advance one assistant turn with `handle.replayTurn()`: ```typescript import { SessionRuntime } from "@pleach/core"; import { ReplayClient, type ReplayRuntimeFacet } from "@pleach/replay"; const runtime = new SessionRuntime({ storage: supabaseAdapter, userId: "user_123", }); // ReplayClient reads only the runtime's `events` facet (iterate + fold). const replay = new ReplayClient({ runtime: runtime as unknown as ReplayRuntimeFacet, }); const handle = await replay.fromEventLog(sessionId, { tenantId: "tenant_xyz", // REQUIRED — replay inherits the runtime's RLS posture }); const turn = await handle.replayTurn(); // advance to the next turn.completed boundary console.log(turn.messageId); // the assistant message that closed the turn console.log(turn.stepResults); // the per-event step results, in fold order console.log(turn.nextState); // the folded HydratedHarnessState after the turn await handle.close(); ``` `replayTurn(messageId?)` loops `step()` to the next `turn.completed` boundary, folding each event through `@pleach/core`'s shared `hydrateFromEvents` reducer. Because incremental `step()` and a full `seek()` re-fold share one reducer, N steps produce a state byte-identical to `seek(N)` — the determinism invariant. To replay against a changed seam (a different provider, an updated prompt contribution, a new safety policy), construct a fresh runtime around the same storage with that seam swapped in, then open a new handle on the new runtime. The runtime has no `withProvider()` / `withSystemPrompt()` mutators — every seam change is a fresh construction. ### Verifying byte-identical replay [#verifying-byte-identical-replay] `createStrictHandleReplay` from `@pleach/replay/strict` is the callable form of the determinism invariant. It opens N independent handles over the same `(chatId, tenantId, window)`, walks each to `done`, and byte-compares the folded state at every step — surfacing the step index where two replays first diverge. ```typescript import { ReplayClient, type ReplayRuntimeFacet } from "@pleach/replay"; import { createStrictHandleReplay } from "@pleach/replay/strict"; const replay = new ReplayClient({ runtime: runtime as unknown as ReplayRuntimeFacet, }); const strict = createStrictHandleReplay({ client: replay }); const verdict = await strict.replay({ chatId: sessionId, tenantId: "tenant_xyz" }); // → { chatId, deterministic: true, steps: 12 } // or { deterministic: false, firstDivergenceAt: 4 } ``` If the verdict is `deterministic: true`, the replay is byte-identical. If it diverges, `firstDivergenceAt` is the step where the chain slipped — walk back through the five contracts in [Determinism](/docs/determinism). ## Regression eval across versions [#regression-eval-across-versions] The fingerprint includes `pleachVersion`. A turn recorded in `1.1.0` will not cache-hit in `1.2.0` automatically — the new version invalidates the bucket. The regression workflow: replay the recorded turn against the new version *without* the cache, capture the new output, and diff. ```typescript // 1. Record under the old version. const recordRuntime = new SessionRuntime({ /* ... */ runtimeMode: "production", // pleachVersion = "1.1.0" }); await recordRuntime.executeMessage(sessionId, prompt); // 2. Re-run under the new version in eval-noncached mode. // (Update the package first; pleachVersion = "1.2.0" now.) const evalRuntime = new SessionRuntime({ /* ... */ runtimeMode: "eval-noncached", }); await evalRuntime.executeMessage(newSessionId, prompt); // 3. Diff the audit ledger rows. const recorded = await ledger.listBySession(sessionId); const evaluated = await ledger.listBySession(newSessionId); const diffs = recorded.map((r, i) => ({ callIndex: i, modelChanged: r.modelId !== evaluated[i]?.modelId, tokenDelta: (evaluated[i]?.tokenUsage.out ?? 0) - r.tokenUsage.out, outcomeChanged: r.outcome.status !== evaluated[i]?.outcome.status, })); ``` The diffs are the regression report. A clean migration produces zero `outcomeChanged: true` rows; otherwise you've found a behavior change to triage. ## Behavioral eval with scorers [#behavioral-eval-with-scorers] Behavioral eval scores model output for quality (helpfulness, accuracy, safety) rather than checking byte equality. Run the turn, then score the output: ```typescript import type { AuditableCall } from "@pleach/core/audit"; interface Scorer { name: string; score(call: AuditableCall, output: string): Promise; } const scorers: Scorer[] = [ { name: "factual-accuracy", async score(call, output) { // Use a judge model to score factual claims: const judge = await judgeModel.evaluate({ prompt, output }); return judge.factualAccuracy; }, }, { name: "concision", async score(call, output) { return output.length < 1000 ? 1.0 : 1000 / output.length; }, }, ]; const synthRow = (await ledger.listBySession(sessionId)) .find((r) => r.callClass === "synthesize"); const scores = await Promise.all( scorers.map(async (s) => ({ name: s.name, score: await s.score(synthRow, synthRow.payload.output as string), })), ); ``` The scorers are pluggable. Build a library matched to your domain — output-format compliance, citation accuracy, refusal calibration — and run them as a post-hoc pass over the recorded ledger. ### The fingerprint pins the comparison [#the-fingerprint-pins-the-comparison] Every score is keyed on `(fingerprint, scorer, version)`. Same input + same model + same scorer = same score. That's what makes "score drift" a real signal rather than measurement noise. ## Forking from a checkpoint [#forking-from-a-checkpoint] Replay can branch from any checkpoint. The substrate ships the checkpoint envelope; the sibling `@pleach/replay` will ship a typed fork API when published. Today's pattern: ```typescript import { SessionRuntime } from "@pleach/core"; // 1. Pick a checkpoint to fork from. const checkpoints = await runtime.listCheckpoints(sessionId); const forkPoint = checkpoints.find((c) => c.stageId === "tool-loop"); // 2. Restore into a fresh session. const forkedSession = await runtime.createSession({ parentSessionId: sessionId, forkFromCheckpoint: forkPoint.id, }); // 3. Replay (or branch with a different prompt). for await (const event of runtime.executeMessage(forkedSession.id, "different prompt")) { // ... } ``` The forked session's audit ledger carries `parentSessionId` and `forkPoint` so the lineage stays queryable through [`LineageTracker`](/docs/lineage). ## Eval CI [#eval-ci] A typical eval-in-CI shape: ```yaml # .github/workflows/eval.yml jobs: regression-eval: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: npm install - run: HARNESS_RUNTIME_MODE=eval-noncached npm run eval:replay - run: npm run eval:diff > eval-report.md - uses: actions/upload-artifact@v4 with: name: eval-report path: eval-report.md ``` The `eval:replay` script runs every fixture against the current build; `eval:diff` compares the new ledger rows against the golden ledger rows committed in the repo. A PR is mergeable when the diff is empty (or all diffs are explained in the PR description). ## What the sibling SKUs will add [#what-the-sibling-skus-will-add] When `@pleach/eval` and `@pleach/replay` ship their first cuts, they layer convenience on top of the primitives above: * **`@pleach/eval`** — fixture format, scorer registry, diff rendering, CI integration. The workflow above expressed as typed config. * **`@pleach/replay`** — `forkFromCheckpoint(runtime, opts)`, divergence-detection on partial replays, replay-mode optimization for cold-cache scenarios. Until they ship, the primitives are enough — the workflow above is the substrate's own test suite, with one fingerprint diff per turn the canonical signal. ## Where to go next [#where-to-go-next]