evalLab

evalLab bakes the canonical wiring of @pleach/core + @pleach/eval + @pleach/replay for research-grade eval runs. The factory constructs a SessionRuntime, hands it to a consumer-supplied EvalSuite factory, and optionally constructs a ReplayClient for deterministic re-derivation of a past run's report from its captured event log.

Best fit: an eval / research lab. The load-bearing constraint is that the SAME suite must reproduce results bit-exactly months later, often for peer review or regulatory audit. The factory bakes the wiring order so the consumer just supplies the model config, the suite cases, and (optionally) the replay-cache factory.

Quickstart

import { evalLab } from "@pleach/recipes/eval-lab";
import { EvalSuite } from "@pleach/eval";
import { ReplayClient } from "@pleach/replay";

const lab = evalLab({
  suiteId: "claude-vs-gpt5-2026-Q3",
  orchestratorConfig: {
    provider: "anthropic",
    model: "claude-sonnet-4-6",
    apiKey: process.env.ANTHROPIC_API_KEY!,
  },
  evalSuiteFactory: (opts) => new EvalSuite(opts),
  replayClientFactory: (runtime) =>
    new ReplayClient({ runtime, tenantId: "lab-001", eventSource: myEventSource }),
});

lab.addCase({
  id: "qa-pubmed-001",
  input: "What gene is implicated in cystic fibrosis?",
  scorer: { type: "expected", expected: "CFTR" },
});

const report = await lab.run();
console.log(`Passed: ${report.summary.passed}/${report.summary.total}`);

// Months later, re-derive the report without re-running the LLM:
const replayed = await lab.replay(report.summary.runId);
console.log(`Identical: ${replayed.summary.passed === report.summary.passed}`);

What it does

The recipe constructs three things in sequence:

1. A SessionRuntime via createPleachRuntime from @pleach/core/runtime, with orchestratorConfig plumbed through host.strategies.orchestratorConfig (mirrors simpleChatbot).
2. An EvalSuite by calling the consumer-supplied evalSuiteFactory({suiteId, runtime}). Calling the factory eagerly at construction lets the consumer call lab.addCase(...) before the first lab.run().
3. (Optional) A ReplayClient by calling the consumer-supplied replayClientFactory(runtime). Without it, lab.replay() throws — research consumers who need deterministic re-derivation must supply one.

lab.run() delegates to suite.run(). lab.replay(runId) delegates to replayClient.fromRun(runId). addCase is an identity pass-through to the suite.

Config reference

type OrchestratorConfigPassthrough = Record<string, unknown>;

interface EvalSuiteLike {
  addCase(c: unknown): void;
  run(): Promise<{
    summary: { runId: string; total: number; passed: number; failed: number };
    cases: readonly unknown[];
  }>;
}

interface ReplayClientLike {
  fromRun(runId: string): Promise<{
    summary: { runId: string; total: number; passed: number; failed: number };
  }>;
}

interface EvalLabConfig extends CreatePleachRuntimeConfig {
  /** Identifier for the suite — surfaces in per-case report rows. */
  suiteId: string;

  /** Provider/orchestrator wiring forwarded to the runtime. */
  orchestratorConfig?: OrchestratorConfigPassthrough;

  /**
   * REQUIRED. Pass `(opts) => new EvalSuite(opts)` from
   * @pleach/eval. The recipe does NOT hard-import the class
   * — that would force a build-time peer dep on the suite
   * class for every consumer of @pleach/recipes.
   */
  evalSuiteFactory: (opts: {
    suiteId: string;
    runtime: SessionRuntime;
  }) => EvalSuiteLike;

  /**
   * Optional. Pass `(runtime) => new ReplayClient({...})` from
   * @pleach/replay. When omitted, lab.replay() throws.
   */
  replayClientFactory?: (runtime: SessionRuntime) => ReplayClientLike;
}

interface EvalLab {
  readonly runtime: SessionRuntime;
  readonly suite: EvalSuiteLike;
  addCase(c: unknown): void;
  run(): ReturnType<EvalSuiteLike["run"]>;
  replay(runId: string): Promise<{
    summary: { runId: string; total: number; passed: number; failed: number };
  }>;
}

Common gotchas

• evalSuiteFactory is REQUIRED. The recipe does NOT hard-import EvalSuite from @pleach/eval — that would force a build-time peer dep on every @pleach/recipes consumer. You supply (opts) => new EvalSuite(opts) and the recipe wires it in. The structural type EvalSuiteLike is the contract.
• replayClientFactory is OPTIONAL but load-bearing for the eval / research lab use case. Without it, lab.replay(runId) throws with a message explaining the contract. Research consumers whose value proposition is reproducible re-derivation must supply one — typically (runtime) => new ReplayClient({ runtime, tenantId, eventSource }).
• The suite is constructed eagerly. evalSuiteFactory fires at evalLab(...) call time, not at first run(). This lets you call addCase(...) before the first run; it also means a factory that throws will surface at construction.
• addCase is an identity pass-through. The recipe does not validate or transform the case payload. The shape is whatever EvalSuite.addCase(c) accepts at the @pleach/eval version you installed.
• Two optional peers, both load-bearing. @pleach/eval is required at runtime for evalSuiteFactory to resolve. @pleach/replay is required only when replayClientFactory is supplied. Install the pair alongside @pleach/recipes.
• Subpath import is load-bearing. Import from @pleach/recipes/eval-lab, not the root barrel.

See also

• @pleach/recipes overview — every recipe in one page.
• @pleach/eval — EvalSuite, scorer types, suite shape.
• @pleach/replay — ReplayClient, fromRun, the deterministic re-derivation contract and the divergence error hierarchy.
• Eval and Replay — the composition pattern this recipe wires.
• Determinism — the substrate guarantee replay relies on.
• enterpriseAgent — composes cleanly alongside evalLab for the enterprise provider-parity validation loop.