@pleach/eval changelog

This page collects the site-content changelog entries that materially touched the @pleach/eval docs surface or named a @pleach/eval symbol. It is not the runtime changelog — the canonical record of @pleach/eval runtime behavior lives in the upstream package CHANGELOG.md and on npmjs.org/package/@pleach/eval.

See the combined site changelog at /docs/changelog/combined for the chronological unified view across every SKU.

Unreleased

Fixed

• Replay no longer promises byte-identical output. A live model can't guarantee it. Eval and replay renames "Verifying byte-identical replay" to "Checking a replay for divergence", and deterministic: true now reads as "no step diverged in this check". The glossary, FAQ, and comparison tables say what ships: replay re-serves the recorded stream and drift raises ReplayDivergenceError.

• EvalSuite accepts cacheBackend and cacheReadPolicy but reads neither (content/docs/eval.mdx). Both are declared on EvalSuiteOptions; the class stores this.opts and reads only suiteId and runtime. The ## Per-suite cache backend section taught them as working knobs. It is now ## Cache configuration and points at the knobs that are live — SessionRuntimeConfig.cacheBackend and runtimeMode: "interactive" | "deterministic", including its fail-closed-without-tenantId rule — and states plainly that the two EvalSuiteOptions fields are accepted and inert in Phase A.

• Stale "planned @pleach/eval" prose corrected — the SKU is shipped at 0.1.0. Channels (two spots) and With Mastra called @pleach/eval "planned" / "when it ships"; it ships at 0.1.0 per Eval, Packages, and Determinism. Ship-status claims are load-bearing, so the stale prose is corrected to match. (The "planned @pleach/eval addition" in Region-pinned agent refers to a specific divergence-detection feature, not the package — left as-is.)

• @pleach/eval cross-mode cache buckets corrected (devharness FINDING 90). The cacheReadPolicy: "cross-mode-readable" note named fictional production / replay / eval-noncached buckets; the real RuntimeMode union (core fingerprint/types.ts) is interactive / headless-eval / headless-replay / headless-job / coding-agent. Corrected to the real one-way read chain interactive → headless-eval → headless-replay.

Added

• Eval Bundled benchmarks section. Documents the @pleach/eval/benchmarks loaders — loadMmluSubset, loadHumanEval, loadSweBenchLite, loadCustomDomain — that turn a public benchmark into EvalCase[].
• Per-recipe deep-dive page — evalLab composes @pleach/eval + @pleach/replay via DI-callback factories so the recipe doesn't take hard build-time peer deps. Rank-2-publish-relevant once eval ships per the upstream rank-2 runbook.

2026-06-08

Changed

• Packages table corrected to real published version — @pleach/eval 0.8.0-alpha.0 (was largely stale "Reserved · placeholder" framing).
• Plugin contract → Sibling SKUs as plugins section retitled and rewritten with per-row real Phase A status — includes @pleach/eval.