# SWE-Bench Lite recipe (/docs/coding-agent/swe-bench-recipe)
[SWE-Bench](https://www.swebench.com/) is a benchmark by Jimenez et
al. (2023) that grades coding agents against real GitHub issues from
popular Python repositories. **SWE-Bench Lite** is a 300-instance
subset selected for tractable single-file edits.
`@pleach/eval/benchmarks/swe-bench` ships a 10-case starter sample as
inline TypeScript constants — enough to scaffold the pipeline against
a model-swap matrix and prove the wiring works. The bundled scorer is
a placeholder; real grading needs a Docker harness OR
[`@pleach/sandbox`](/docs/sandbox)-driven code execution with a
judge. This page walks both.
## Honest scope [#honest-scope]
The bundled scorer returns `score: 0` with an explanation prompting
you to wire a real harness. SWE-Bench grading requires applying a
model-generated patch to the repo, running the repo's test suite, and
comparing test outcomes to the "before patch" baseline. None of that
is shipped in `@pleach/eval` — by design, the SKU is Docker-free and
vendor-neutral.
You wire the harness. The bundled sample is for pipeline scaffolding,
not a SWE-Bench Lite official run.
Cite the upstream dataset when reporting results; do not claim the
starter sample is a SWE-Bench Lite official run.
## Load the sample [#load-the-sample]
```ts
import { loadSweBenchLite } from "@pleach/eval/benchmarks/swe-bench";
const cases = await loadSweBenchLite({ maxCases: 5 });
console.log(cases.length); // 5
console.log(cases[0].id); // "swebench-lite/django__django-11099"
```
Each case is an `EvalCase` of kind `"scored"`, with `input` containing:
| Field | Shape | Example |
| ------------------ | ---------------------- | ------------------------------------------------ |
| `repo` | `string` | `"django/django"` |
| `issueNumber` | `number` | `11099` |
| `problemStatement` | `string` (paraphrased) | `"UsernameValidator allows trailing newline..."` |
| `hints` | `string` (paraphrased) | `"Look at django/contrib/auth/validators.py..."` |
| `source` | `string` | `"SWE-Bench Lite — Jimenez et al. 2023 (MIT)"` |
The bundled cases reference real upstream repos + issue numbers; the
`problemStatement` and `hints` fields are paraphrased to keep the
bundle small. The 10 cases cover Django, SymPy, scikit-learn,
matplotlib, pytest, requests, and Flask.
## Compose with `evalLab` [#compose-with-evallab]
The `evalLab` recipe from `@pleach/recipes/eval-lab` wires a runtime
* eval suite + (optional) replay client trio for research
reproducibility. SWE-Bench fits the shape:
```ts
import { evalLab } from "@pleach/recipes/eval-lab";
import { loadSweBenchLite } from "@pleach/eval/benchmarks/swe-bench";
import { EvalSuite } from "@pleach/eval";
const lab = evalLab({
suiteId: "swe-bench-lite-claude-sonnet-4-6",
orchestratorConfig: {
provider: "anthropic",
model: "claude-sonnet-4-6",
apiKey: process.env.ANTHROPIC_API_KEY!,
},
evalSuiteFactory: ({ suiteId, runtime }) =>
new EvalSuite({ suiteId, runtime }),
});
const cases = await loadSweBenchLite({ maxCases: 10 });
for (const c of cases) lab.addCase(c);
const report = await lab.run();
console.log(`${report.summary.passed}/${report.summary.total}`);
```
The lab returns immediately; the suite is constructed eagerly so the
consumer can add cases before the first `run()`. `evalSuiteFactory`
avoids a hard build-time peer-dep on the suite class — `@pleach/eval`
is an OPTIONAL peer of `@pleach/recipes`.
## Run across a model matrix [#run-across-a-model-matrix]
`runMatrixBatch` from `@pleach/eval` swaps the model across N
variants and runs the same cases against each, surfacing per-variant
per-case outcomes:
```ts
import { runMatrixBatch, createModelSwapSubjects } from "@pleach/eval";
import { loadSweBenchLite } from "@pleach/eval/benchmarks/swe-bench";
const cases = await loadSweBenchLite({ maxCases: 10 });
const subjects = createModelSwapSubjects({
models: ["claude-sonnet-4-6", "claude-opus-4-7", "claude-haiku-4-6"],
buildSubject: (model) => ({
id: model,
run: async (scenario) => runCaseAgainstModel(model, scenario),
}),
});
const matrixReport = await runMatrixBatch({
subjects,
scenarios,
scorers,
});
```
The matrix report carries one `RunCaseDivergence` row per `(subject, case)`
pair. Three subjects × ten cases = thirty rows.
## Surface divergence with `DivergenceReporter` [#surface-divergence-with-divergencereporter]
`createDivergenceReporter` projects the matrix report into a
human-readable summary + per-case breakdown + heatmap-ready data:
```ts
import { createDivergenceReporter } from "@pleach/eval";
const reporter = createDivergenceReporter({ runs });
console.log(reporter.summary());
// {
// subjectCount: 3,
// caseCount: 10,
// agreementRate: 0.6, // 6 of 10 cases all 3 subjects agreed
// divergenceRate: 0.4,
// }
for (const row of reporter.perCase()) {
if (row.divergences.length > 0) {
console.log(row.caseId, row.divergences);
}
}
// For a downstream chart:
const heatmap = reporter.export("markdown");
```
The reporter is pure — given the same matrix report, it returns the
same summary and per-case rows. Pair with [`@pleach/replay`](/docs/replay)
to re-derive the matrix report from a captured event log months later
without re-running the LLM.
## Wiring a real scorer [#wiring-a-real-scorer]
The bundled placeholder scorer returns `score: 0`. To grade
SWE-Bench Lite properly:
### Option A — Docker harness (canonical) [#option-a--docker-harness-canonical]
Apply the model-generated patch to a checkout of the target repo at
`baseCommit`, run the repo's test suite inside a Docker container,
diff the test outcomes against the pre-patch baseline. Pass-to-fail
transitions on the issue's target tests = success.
This is what the upstream
[SWE-Bench evaluation harness](https://github.com/princeton-nlp/SWE-bench)
does. The harness is a separate concern from `@pleach/eval` and is
not bundled.
```ts
import { loadSweBenchLite } from "@pleach/eval/benchmarks/swe-bench";
const cases = await loadSweBenchLite();
const cases_with_real_scorer = cases.map((c) => ({
...c,
scorer: {
kind: "custom",
fn: async (actualPatch, ctx) => {
// `EvalCase.input` is typed `string | Record`;
// narrow to the SWE-Bench row shape before reading fields.
const input = c.input as { repo: string; issueNumber: number };
// Apply patch + run tests in your Docker harness.
const result = await yourDockerHarness.grade({
repo: input.repo,
issueNumber: input.issueNumber,
patch: actualPatch,
});
return {
score: result.passToFail ? 1 : 0,
explanation: result.summary,
};
},
},
}));
```
### Option B — `@pleach/sandbox`-driven judge [#option-b--pleachsandbox-driven-judge]
For a lighter-weight option, drive the test run through a sandbox
provider — the same provider your coding agent uses. Apply the patch
via `apply_diff`, run the suite via `run_tests`, judge the outcome:
```ts
import { createSandboxComposite } from "@pleach/coding-agent/sandbox";
import { createApplyDiffTool, createRunTestsTool } from "@pleach/coding-agent/tools";
const session = await provider.acquire();
const composite = createSandboxComposite({ session });
const cases_with_sandbox_scorer = cases.map((c) => ({
...c,
scorer: {
kind: "custom",
fn: async (actualPatch, ctx) => {
// `EvalCase.input` is typed `string | Record`;
// narrow to the SWE-Bench row shape before reading fields.
const input = c.input as { repo: string; baseCommit: string };
// 1. Clone repo at baseCommit.
await composite.gitClone(`https://github.com/${input.repo}.git`, {
ref: input.baseCommit,
});
// 2. Apply the model's patch via apply_diff tool.
const apply = await applyDiff.execute(
{ path: ".", diff: actualPatch },
{ toolCallId: "swe-bench-grade" },
);
if (apply.conflicts.length > 0) {
return { score: 0, explanation: `patch conflicts: ${apply.conflicts.join("; ")}` };
}
// 3. Run tests via run_tests tool.
const tests = await runTests.execute(
{ command: "pytest -xvs", cwd: input.repo.split("/")[1] },
{ toolCallId: "swe-bench-grade" },
);
return {
score: tests.failed === 0 && (tests.passed ?? 0) > 0 ? 1 : 0,
explanation: `exit ${tests.exitCode}; passed ${tests.passed} failed ${tests.failed}`,
};
},
},
}));
```
This is faster to set up than option A (no Docker daemon required)
but less faithful to the upstream SWE-Bench evaluation methodology
— in particular, network access, dependency installation, and test
isolation are vendor-dependent in the sandbox case.
## What `@pleach/eval` does NOT ship [#what-pleacheval-does-not-ship]
* A Docker harness.
* The pre-computed `test_patch` + `baseCommit` + `expected_pass_to_fail`
metadata that the official SWE-Bench dataset carries. The bundled
sample paraphrases the `problemStatement`; full upstream metadata
is left to the consumer to fetch from
[`princeton-nlp/SWE-bench_Lite`](https://huggingface.co/datasets/princeton-nlp/SWE-bench_Lite).
* A full 300-case Lite corpus. The 10-case starter is for pipeline
scaffolding; production runs should load the full corpus from
upstream.
## Where to go next [#where-to-go-next]