# Eval · Parity (/docs/eval/parity)



`@pleach/eval/parity` ships two parity primitives that answer
**orthogonal questions** about output stability:

| Question                                                | Primitive                           |
| ------------------------------------------------------- | ----------------------------------- |
| "Is my swarm fan-out deterministic across iterations?"  | [`runSwarmParity`](#runswarmparity) |
| "Does swapping the model materially change the output?" | [`runParitySuite`](#runparitysuite) |

Both share `editDistance` from
`packages/eval/src/divergence/distance.ts` as the underlying string
comparison primitive, and both lean on the same divergence aggregator
shape. The split exists because the **shape of the input** is
genuinely different: `runSwarmParity` consumes `SwarmRunResult[]`
(N iterations of one configuration), while `runParitySuite` consumes
`ParitySuiteRun[]` (M configurations on K scenarios).

## `runSwarmParity` [#runswarmparity]

The within-swarm stability runner. Given a swarm runner that produces
`SwarmRunResult` from a scenario, `runSwarmParity` invokes the runner
`iterations` times per scenario (default 3) and reports per-scenario
divergence + outlier session IDs.

```ts
import { runSwarmParity } from "@pleach/eval/parity";

const report = await runSwarmParity({
  runner: {
    runScenario: async (scenario) => {
      // Spin up a swarm fan-out from the scenario; collect sub-agent
      // outputs in BFS order.
      return await mySwarmEntryPoint(scenario);
    },
  },
  scenarios: [scenarioA, scenarioB, scenarioC],
  iterations: 3, // default
});

console.log(report.scenarios[0].divergence.outlierSessionIds);
```

### Iteration semantics [#iteration-semantics]

Two iterations is the practical minimum (anything less leaves "outlier"
undefined); three is the default to surface a stable majority vs a
single outlier. Consumers may pass higher `iterations` for
high-variance workloads at proportional cost.

### Output shape [#output-shape]

```ts
interface SwarmParityReport {
  readonly scenarioCount: number;
  readonly iterations: number;
  readonly scenarios: ReadonlyArray<{
    readonly scenarioLabel: string;
    readonly divergence: SwarmDivergenceResult; // .outlierSessionIds lives here
    // ...
  }>;
  readonly meanPairwiseDistance: number;
  readonly outlierSessionIds: ReadonlyArray<string>; // union across scenarios
}
```

`computeSwarmDivergence` reduces N iterations into pairwise distance +
the outlier-session-id list. Outliers are flagged at a 1.5× threshold
against the cohort mean — the structural minimum for a 3-config
single-outlier cohort.

## `runParitySuite` [#runparitysuite]

The across-configuration divergence runner. Given M `ParitySuiteRun[]`
(each with its own `id` and `runner(scenario)`) and a list of scenarios,
`runParitySuite` runs every configuration over every scenario and
emits a scenario-keyed report.

```ts
import { runParitySuite } from "@pleach/eval/parity";

const report = await runParitySuite({
  runs: [
    { id: "claude-opus", runner: scenarioRunnerOpus },
    { id: "gpt-5", runner: scenarioRunnerGpt },
    { id: "gemini-2", runner: scenarioRunnerGemini },
  ],
  scenarios: [scenarioA, scenarioB, scenarioC],
  scenarioLabels: ["math-word-problem", "code-refactor", "summarization"],
  tolerance: { distance: 50 }, // characters of editDistance per scenario
});

console.log(report.divergentScenarioIds);
// ["code-refactor"] — the scenarios that exceeded tolerance
```

### Tolerance semantics [#tolerance-semantics]

`tolerance.distance` sets the maximum `editDistance` between any two
configurations' outputs for one scenario before that scenario is
flagged as divergent. Default: `0` (strict — any character-level
difference flags). Per-scenario `ScenarioParity.divergent` is `true`
when **any** pairwise distance exceeds the threshold.

`report.divergentScenarioIds` is the union of flagged scenario IDs —
the top-level summary the consumer typically pipes into a CI gate.

### Output shape [#output-shape-1]

```ts
interface ParityReport {
  readonly runCount: number;
  readonly scenarioCount: number;
  readonly scenarios: ReadonlyArray<ScenarioParity>;
  readonly meanDistance: number;     // cohort mean
  readonly maxDistance: number;      // cohort worst pair
  readonly divergentScenarioIds: ReadonlyArray<string>;
}

interface ScenarioParity {
  readonly scenarioId: string;
  readonly outputs: ReadonlyArray<{ runId: string; output: string }>;
  readonly pairwise: ReadonlyArray<DivergencePair>;
  readonly maxDistance: number;
  readonly meanDistance: number;
  readonly divergent: boolean;
}
```

### Outlier detection — the 1.3× threshold [#outlier-detection--the-13-threshold]

The cohort outlier set (the configurations whose mean per-scenario
distance to peers exceeds a multiplier of the cohort mean) is
exposed via the shared `computeDivergence` aggregator and surfaced as
`outlierRunIds`. The multiplier is &#x2A;*1.3×** on the non-swarm runner.

The brief on choosing 1.3× rather than the swarm runner's 1.5×: a 3-config
single-outlier cohort with `editDistance` distribution `[10, 10, 30]`
has a cohort mean of `(10 + 10 + 30) / 3 ≈ 16.67`; the outlier's
per-peer mean is `30`, and `30 / 16.67 ≈ 1.8`. A 1.5× multiplier is
the structural minimum to flag this cohort's outlier; 1.3× is the
more sensitive setting for configuration-level parity, where a single character-level
divergence already matters more than swarm-level row-level noise.

## Sequential dispatch by design [#sequential-dispatch-by-design]

Both runners dispatch sequentially within and across iterations.
Consumers own concurrency — wrap `runScenario` / `runner` in
`Promise.all` if you want parallel fan-out, but be aware that:

* **Deterministic test ordering** is preserved by sequential dispatch.
* **Rate-limit budget** is easier to predict — N iterations × M
  scenarios maps directly to N × M provider calls.
* **Replay safety** is straightforward — the dispatch order matches
  the recorded event-log order.

Consumers who need parallel dispatch typically wrap the runner with
their own concurrency adapter and accept that the report's
ordering-sensitive fields become substrate-dependent.

## Cited source [#cited-source]

* `packages/eval/src/parity/runSwarmParity.ts` — within-swarm runner.
* `packages/eval/src/parity/runParitySuite.ts` — across-config runner.
* `packages/eval/src/parity/swarmDivergenceMetrics.ts` —
  `computeSwarmDivergence` aggregator.
* `packages/eval/src/parity/divergenceMetrics.ts` —
  `computeDivergence` aggregator.
* `packages/eval/src/divergence/distance.ts` — shared `editDistance`
  primitive.
* `packages/eval/src/parity/index.ts` — barrel re-export.

## Where to go next [#where-to-go-next]

<Cards>
  <Card title="Eval" href="/docs/eval" description="The full @pleach/eval surface — divergence reporting, benchmark loaders, cost estimator." />

  <Card title="Subagents" href="/docs/subagents" description="The canonical spawn substrate that runSwarmParity consumes." />

  <Card title="Swarm agent" href="/docs/swarm-agent" description="End-to-end recipe for swarm fan-out with deterministic baseline + outlier review." />

  <Card title="Replay" href="/docs/replay" description="StrictReplay pairs naturally with parity reports — replay the flagged scenarios with cache adapters." />
</Cards>