# evalLab (/docs/recipes/eval-lab)



`evalLab` bakes the canonical wiring of `@pleach/core` +
`@pleach/eval` + `@pleach/replay` for research-grade eval
runs. The factory constructs a `SessionRuntime`, hands it to
a consumer-supplied `EvalSuite` factory, and optionally
constructs a `ReplayClient` for deterministic re-derivation
of a past run's report from its captured event log.

Best fit: **an eval / research lab**. The
load-bearing constraint is that the SAME suite must
reproduce results bit-exactly months later, often for peer
review or regulatory audit. The factory bakes the wiring
order so the consumer just supplies the model config, the
suite cases, and (optionally) the replay-cache factory.

## Quickstart [#quickstart]

```ts
import { evalLab } from "@pleach/recipes/eval-lab";
import { EvalSuite } from "@pleach/eval";
import { ReplayClient } from "@pleach/replay";

const lab = evalLab({
  suiteId: "claude-vs-gpt5-2026-Q3",
  orchestratorConfig: {
    provider: "anthropic",
    model: "claude-sonnet-4-6",
    apiKey: process.env.ANTHROPIC_API_KEY!,
  },
  evalSuiteFactory: (opts) => new EvalSuite(opts),
  replayClientFactory: (runtime) =>
    new ReplayClient({ runtime, tenantId: "lab-001", eventSource: myEventSource }),
});

lab.addCase({
  id: "qa-pubmed-001",
  input: "What gene is implicated in cystic fibrosis?",
  scorer: { type: "expected", expected: "CFTR" },
});

const report = await lab.run();
console.log(`Passed: ${report.summary.passed}/${report.summary.total}`);

// Months later, re-derive the report without re-running the LLM:
const replayed = await lab.replay(report.summary.runId);
console.log(`Identical: ${replayed.summary.passed === report.summary.passed}`);
```

## What it does [#what-it-does]

The recipe constructs three things in sequence:

1. A `SessionRuntime` via `createPleachRuntime` from
   `@pleach/core/runtime`, with `orchestratorConfig`
   plumbed through `host.strategies.orchestratorConfig`
   (mirrors `simpleChatbot`).
2. An `EvalSuite` by calling the consumer-supplied
   `evalSuiteFactory({suiteId, runtime})`. Calling the
   factory eagerly at construction lets the consumer call
   `lab.addCase(...)` before the first `lab.run()`.
3. (Optional) A `ReplayClient` by calling the
   consumer-supplied `replayClientFactory(runtime)`. Without
   it, `lab.replay()` throws — research consumers who need
   deterministic re-derivation must supply one.

`lab.run()` delegates to `suite.run()`. `lab.replay(runId)`
delegates to `replayClient.fromRun(runId)`. `addCase` is an
identity pass-through to the suite.

## Config reference [#config-reference]

```ts
type OrchestratorConfigPassthrough = Record<string, unknown>;

interface EvalSuiteLike {
  addCase(c: unknown): void;
  run(): Promise<{
    summary: { runId: string; total: number; passed: number; failed: number };
    cases: readonly unknown[];
  }>;
}

interface ReplayClientLike {
  fromRun(runId: string): Promise<{
    summary: { runId: string; total: number; passed: number; failed: number };
  }>;
}

interface EvalLabConfig extends CreatePleachRuntimeConfig {
  /** Identifier for the suite — surfaces in per-case report rows. */
  suiteId: string;

  /** Provider/orchestrator wiring forwarded to the runtime. */
  orchestratorConfig?: OrchestratorConfigPassthrough;

  /**
   * REQUIRED. Pass `(opts) => new EvalSuite(opts)` from
   * @pleach/eval. The recipe does NOT hard-import the class
   * — that would force a build-time peer dep on the suite
   * class for every consumer of @pleach/recipes.
   */
  evalSuiteFactory: (opts: {
    suiteId: string;
    runtime: SessionRuntime;
  }) => EvalSuiteLike;

  /**
   * Optional. Pass `(runtime) => new ReplayClient({...})` from
   * @pleach/replay. When omitted, lab.replay() throws.
   */
  replayClientFactory?: (runtime: SessionRuntime) => ReplayClientLike;
}

interface EvalLab {
  readonly runtime: SessionRuntime;
  readonly suite: EvalSuiteLike;
  addCase(c: unknown): void;
  run(): ReturnType<EvalSuiteLike["run"]>;
  replay(runId: string): Promise<{
    summary: { runId: string; total: number; passed: number; failed: number };
  }>;
}
```

## Common gotchas [#common-gotchas]

* **`evalSuiteFactory` is REQUIRED.** The recipe does NOT
  hard-import `EvalSuite` from `@pleach/eval` — that would
  force a build-time peer dep on every `@pleach/recipes`
  consumer. You supply
  `(opts) => new EvalSuite(opts)` and the recipe wires it
  in. The structural type `EvalSuiteLike` is the contract.
* **`replayClientFactory` is OPTIONAL but load-bearing for
  the eval / research lab use case.** Without it, `lab.replay(runId)` throws
  with a message explaining the contract. Research
  consumers whose value proposition is reproducible
  re-derivation must supply one — typically
  `(runtime) => new ReplayClient({ runtime, tenantId,
  eventSource })`.
* **The suite is constructed eagerly.** `evalSuiteFactory`
  fires at `evalLab(...)` call time, not at first
  `run()`. This lets you call `addCase(...)` before the
  first run; it also means a factory that throws will
  surface at construction.
* **`addCase` is an identity pass-through.** The recipe
  does not validate or transform the case payload. The
  shape is whatever `EvalSuite.addCase(c)` accepts at the
  `@pleach/eval` version you installed.
* **Two optional peers, both load-bearing.** `@pleach/eval`
  is required at runtime for `evalSuiteFactory` to
  resolve. `@pleach/replay` is required only when
  `replayClientFactory` is supplied. Install the pair
  alongside `@pleach/recipes`.
* **Subpath import is load-bearing.** Import from
  `@pleach/recipes/eval-lab`, not the root barrel.

## See also [#see-also]

* [`@pleach/recipes` overview](/docs/recipes-pleach-recipes)
  — every recipe in one page.
* [`@pleach/eval`](/docs/eval) — `EvalSuite`,
  scorer types, suite shape.
* [`@pleach/replay`](/docs/replay) — `ReplayClient`,
  `fromRun`, the deterministic re-derivation contract and
  the divergence error hierarchy.
* [`Eval and Replay`](/docs/eval-and-replay) — the
  composition pattern this recipe wires.
* [`Determinism`](/docs/determinism) — the substrate
  guarantee replay relies on.
* [`enterpriseAgent`](/docs/recipes/enterprise-agent) —
  composes cleanly alongside `evalLab` for the enterprise
  provider-parity validation loop.