# @pleach/eval (/docs/eval) How the garden grew this season — fixed seeds, measured shoots, comparable rows. `@pleach/eval` is the evaluation harness for `@pleach/core` agent runtimes. The package ships an `EvalSuite` class that accepts cases as a discriminated union, four built-in scorers, three report formatters (JSON, Markdown, JUnit XML), and a Phase B `EvalRuntime` factory with live bodies for batch execution, report rendering, structural `diffReports`, and statistical `compareScored` (bootstrap + Welch's t-test). Coupling to `@pleach/replay` is opt-in via dependency injection — the two SKUs install and adopt independently. This page is the SKU reference. For the DIY workflow using `@pleach/core` primitives directly today (without this SKU), see [Eval and replay](/docs/eval-and-replay). ## Install [#install] ```bash npm install @pleach/eval @pleach/core ``` ```bash pnpm add @pleach/eval @pleach/core ``` ```bash bun add @pleach/eval @pleach/core ``` `@pleach/core` is a peer dependency. ```typescript import { EvalSuite } from "@pleach/eval"; import { tokenF1 } from "@pleach/eval/scorers"; import { formatMarkdown } from "@pleach/eval/report"; import { NoopReplayClient } from "@pleach/eval/replay"; ``` The import names above are illustrative. The authoritative export shape lives in the package README on [npm](https://www.npmjs.com/package/@pleach/eval) — check it when pinning a version, since the surface evolves between Phase A and the 1.0 cut. ## The `EvalSuite` class [#the-evalsuite-class] `EvalSuite` is the entry point. Construct it with a `suiteId` (the identifier that surfaces in reports), register cases, then call `run()` to produce an `EvalReport`. ```typescript import { EvalSuite } from "@pleach/eval"; const suite = new EvalSuite({ suiteId: "my-eval" }); suite.register({ kind: "expected", id: "greeting", input: "hello", expected: "hello", }); const report = await suite.run(); console.log(report.summary); // { total: 1, passed: 1, failed: 0, errored: 0 } ``` Cases run in registration order. The report carries a stable `runId`, the start and completion timestamps, every case result, and a summary count. The constructor takes a `runtime` (any `EvalSessionRuntimeLike` — a real `SessionRuntime` from `@pleach/core` satisfies the shape), a `cacheBackend` (defaults per-suite — see below), and a `cacheReadPolicy` (`"strict-mode"` for research replication, `"cross-mode-readable"` for graded coursework). The `runtime` is optional in Phase A so the surface unit-tests against a mock; in production callers always provide a real runtime. ## Case shapes [#case-shapes] `EvalCase` is a discriminated union with three shapes. There is no global registry — scorers ride inline with the case. ### `expected` — pass/fail equality [#expected--passfail-equality] A string or RegExp equality check. ```typescript suite.register({ kind: "expected", id: "section-cite", input: "the model output", expected: /section 4\.\d+/, }); ``` ### `scored` — numeric score from a scorer [#scored--numeric-score-from-a-scorer] A scorer returns a score in `[0, 1]`. Reference a built-in by id or ship a custom function inline. ```typescript suite.register({ kind: "scored", id: "f1-against-golden", input: { expected: "the quick brown fox" }, scorer: { kind: "builtin", id: "tokenF1" }, }); suite.register({ kind: "scored", id: "length-bound", input: "...", scorer: { kind: "custom", fn: async (actual) => ({ score: actual.length > 100 ? 1 : 0, explanation: `length=${actual.length}`, }), }, }); ``` A `scored` case passes when its score is `>= 1`. Anything lower is a fail, but the numeric score still rolls into the report's mean. ### `judged` — LLM-as-judge ensemble [#judged--llm-as-judge-ensemble] A list of judge models that score the actual output. Each judge carries a model id, a rubric prompt, and a score schema (`"binary"`, `"likert-5"`, or `"numeric"`). ```typescript suite.register({ kind: "judged", id: "summary-accuracy", input: "summarize this paper: ...", judges: [ { model: "anthropic/claude-opus-4-7", rubric: "Rate the summary's factual accuracy on a 1-5 Likert scale.", scoreSchema: "likert-5", }, ], }); ``` Phase A ships the contract; judge dispatch is stubbed. Judged cases return a placeholder score and an explanation noting the stub. The Phase B landing wires dispatch through the runtime's provider seam so judges route through `@pleach/gateway` when present. ## The four built-in scorers [#the-four-built-in-scorers] Exported from `@pleach/eval/scorers`. All four implement the `EvalScorerFn` contract — `(actual, ctx) => EvalScore` — so they work as both inline values and `EvalScorerRef` references. | Scorer | Input expectation | Score | | ------------------- | ------------------------------------------ | --------------------------------------------- | | `exactMatch` | `expected` string in case input | 1 iff `actual === expected` | | `substringContains` | `expected` string in case input | 1 iff `actual.includes(expected)` | | `regexMatch` | `pattern` + optional `flags` in case input | 1 iff the regex tests true | | `tokenF1` | `expected` string in case input | Token-level F1 (whitespace-split, lowercased) | `exactMatch`, `substringContains`, and `tokenF1` read the expected value from either a bare string input or an `{ expected }` field on a record input. `regexMatch` reads `{ pattern, flags? }` (or treats a bare string input as the pattern with no flags). When the expected value is missing, the scorer returns score 0 with an explanation naming the missing field — no thrown exceptions for misconfigured cases. `tokenF1` returns the F1 over the token multiset: `2PR / (P + R)`, where P is overlap divided by `actual` token count and R is overlap divided by `expected` token count. Both sides empty score 1 by convention; one side empty scores 0. ## The three report formats [#the-three-report-formats] Exported from `@pleach/eval/report`. The same `EvalReport` feeds all three formatters — pick one (or several) depending on where the report lands. ```typescript import { formatJSON, formatMarkdown, formatJUnitXML } from "@pleach/eval/report"; const report = await suite.run(); console.log(formatJSON(report)); // pretty-printed JSON console.log(formatMarkdown(report)); // human-readable summary console.log(formatJUnitXML(report)); // CI consumer ``` `formatJSON` sorts case results by `caseId` and emits stable key ordering — two runs against identical input produce byte-identical JSON output, which makes the formatter usable as a golden-file target. Two-space indent matches `jq` defaults. `formatMarkdown` emits a summary block plus a per-case table with status, score, and explanation. Pipe character escaping is handled. `formatJUnitXML` produces a single `` element with one `` per eval case. Errored cases include ``, failed cases include ``. The shape matches the `junit-xml` convention used by `mocha-junit-reporter` and `jest-junit`, so the output plugs into GitHub Actions, GitLab CI, and Jenkins without a custom reporter. ## Replay coupling via constructor DI [#replay-coupling-via-constructor-di] `@pleach/eval` does not import `@pleach/replay`. The two SKUs adopt independently and couple through a structural contract — `EvalReplayClient` — that consumers (or `@pleach/replay` itself) implement. ```typescript import { EvalSuite, type EvalReplayClient } from "@pleach/eval"; import { createReplayRuntime } from "@pleach/replay"; const suite = new EvalSuite({ suiteId: "replay-eval", runtime }); // @pleach/replay's turn-granular runtime, adapted to eval's structural // `EvalReplayClient` contract (`replay(chatId) => { output }`). const tenantId = "tenant_xyz"; const replayRuntime = createReplayRuntime({ tenantId, sessionRuntime: runtime }); const replayClient: EvalReplayClient = { async replay(chatId) { const { state } = await replayRuntime.replayTurn({ chatId, tenantId }); return { output: String(state ?? "") }; }, }; suite.setReplayClient(replayClient); const report = await suite.replay("chat_01HKXJ..."); ``` `replay(chatId)` reconstructs the turn's final output via the injected client and runs every registered case against it. Without a client, `replay()` throws with a message that names the missing DI. `@pleach/eval/replay` ships two test stubs: * `NoopReplayClient` — returns empty output for any chat id. Useful when unit-testing the `replay()` wiring without dragging in the real replay package. * `StaticReplayClient` — returns a canned output for any chat id. Useful when unit-testing scorer dispatch against a known output. ```typescript import { NoopReplayClient, StaticReplayClient } from "@pleach/eval/replay"; suite.setReplayClient(new StaticReplayClient("the canned output")); ``` The stubs ship in the same subpath as the contract, so consumers that only need the structural type — not the real replay client — don't pick up a transitive dependency. ## Per-suite cache backend [#per-suite-cache-backend] The cache backend defaults to a fresh in-memory backend per `EvalSuite` instance. That isolation is the load-bearing piece: parallel suites in the same process can't poison each other's cache, and a `run()` against a deterministic runtime produces byte-identical reports on repeat. Override when you want the suite to share state with a production runtime: ```typescript import { createMemoryCacheBackend } from "@pleach/core/cache"; const suite = new EvalSuite({ suiteId: "shared-cache-eval", runtime, cacheBackend: createMemoryCacheBackend({ maxEntries: 10_000 }), cacheReadPolicy: "strict-mode", }); ``` `cacheReadPolicy: "strict-mode"` is the research-replication setting — a cache miss in strict mode is an error, not a silent fallthrough. `"cross-mode-readable"` is the graded-coursework setting where reads can cross between `interactive` / `headless-eval` / `headless-replay` buckets. See [Cache](/docs/cache) for the policy contract. ## Phase A status [#phase-a-status] The package shipped at `0.1.0` as Phase A of the SKU. The Phase A `EvalSuite` contract is in place; two pieces of behavior are stubbed and land in a follow-up: * **Judge dispatch.** Judged cases record the ensemble shape but do not invoke any provider. Judge dispatch wires through `runtime.providers` so judges route through `@pleach/gateway` when it's configured. * **Default actual-output resolver.** Phase A's default resolver echoes the case input back as the actual output. Production callers override via `setActualResolver(resolver)` to drive a real runtime turn. The default keeps the surface unit-testable; nothing about the report shape changes when the resolver is swapped. Both behaviors are tracked against the package's published changelog on npm; pin a version with the change you need. ## Phase B runtime — `createEvalRuntime` [#phase-b-runtime--createevalruntime] The Phase B `EvalRuntime` factory (`createEvalRuntime`) ships alongside the Phase A surface and exposes the batch / report / comparison surface used for cross-config and cross-run analysis. At the current cut: | `EvalRuntime` method | Status | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | `runScenario(input)` | Live — validates input + emits a stub `EvalScenarioOutput` (real LLM invocation routes through `harnessRuntime` in a follow-up) | | `runBatch(input)` | Live — iterates `configs × rows`, dispatches each cell via `runScenario`, aggregates per-config + global cost rolls | | `produceEvalReport(input)` | Live — renders a versioned report in JSON / Markdown / CSV format | | `recordOutcome(input)` | Live — appends to an in-memory outcomes ledger (persistent storage via `@pleach/replay` cache is a follow-up) | | `diffReports(prior, next)` | Live — pure-function deterministic structural diff over two `EvalReportOutput` instances | | `compareScored(prior, next, opts?)` | Live — bootstrap resampling + Welch's t-test over two scored runs | | `runStreaming(input, callbacks)` | Live — streaming variant of `runBatch` with per-cell callbacks | ### `diffReports(prior, next)` [#diffreportsprior-next] Pure-function structural diff between two `EvalReportOutput` instances. Both inputs must carry the same `version` and `format: "json"` (the canonical JSON format is parsed and walked cell-by-cell). The returned `EvalReportDiff` carries per-cell status (`added`, `removed`, `changed`, `unchanged`) and a summary roll-up. Use it to verify two pinned report snapshots match, or to surface what changed between a pre- and post-refactor batch. ### `compareScored(prior, next, opts?)` [#comparescoredprior-next-opts] Statistical comparison of two scored eval runs. Pairs cells by `(config, row)` identity, then runs bootstrap resampling and Welch's t-test on the paired score deltas — paired-sample comparison with unequal variances (Welch's t-test is the unequal-variance generalization of Student's t-test; bootstrap resampling gives a non-parametric confidence interval that holds without distributional assumptions on the score). The returned `EvalComparisonResult` carries the bootstrap CI, the t-test statistic + p-value, and per-row pairing diagnostics so a caller can reject "the new config is statistically better" claims that fail the significance bar. Pair this with [`@pleach/replay`](/docs/replay) when the eval batches you're comparing are themselves derived from chat replays — `@pleach/eval` consumes `EvalReportOutput`s, and `@pleach/replay` is the substrate that produces deterministic outputs to score. ## Where to go next [#where-to-go-next]