# Cache & memoization (/docs/cache)



The runtime memoizes prepared LLM inputs through a `CacheBackend`
contract — expanded system prompts, normalized message arrays, the
work the seam does *before* it hands a request to the provider. This
is runtime-side caching. It's distinct from
[provider-side prompt caching](/docs/prompt-caching), which is the
Anthropic/OpenAI server-side prefix-reuse mechanism the substrate
also rides on.

Both layers cut latency. Only this one is under the substrate's
direct control, which is why the contract is short and the
correctness rules are explicit.

<SourceMeta subpath="@pleach/core/cache" source="{ label: &#x22;src/cache/&#x22;, href: &#x22;https://github.com/pleachhq/core/tree/main/src/cache&#x22; }" />

## `CacheBackend` contract [#cachebackend-contract]

One interface; three methods. Every implementation conforms.

```typescript
interface CacheBackend {
  get(key: Fingerprint): Promise<CacheEntry | null>;
  set(key: Fingerprint, value: CacheEntry): Promise<void>;
  metricsSnapshot(): CacheMetricsSnapshot;
}
```

Two contract rules:

1. **`set` is fire-and-forget.** The seam invokes it without
   awaiting in the hot path. A backend that throws synchronously
   breaks the turn — see [What not to do](#what-not-to-do).
2. **`metricsSnapshot` is O(1).** Counters update on every
   mutation; the snapshot reads them, never recomputes.

The returned `CacheMetricsSnapshot` carries hit/miss counters,
entry count, and byte usage:

```typescript
interface CacheMetricsSnapshot {
  readonly hits: number;
  readonly misses: number;
  readonly entryCount: number;
  readonly sizeBytes: number;
}
```

A miss returns `null` rather than throwing. A backend outage
under the `best-effort` policy degrades to a miss too — the seam
falls through to the provider call instead of failing the turn.

### `CacheGetMode` vs `CacheReadPolicy` [#cachegetmode-vs-cachereadpolicy]

Two orthogonal policies, both on the read path. Don't conflate them.

| Policy            | Type                                     | Scope                        | What it governs                                                                           |
| ----------------- | ---------------------------------------- | ---------------------------- | ----------------------------------------------------------------------------------------- |
| `CacheGetMode`    | `"strict-fail" \| "best-effort"`         | Per-call (passed to `get()`) | Failure policy when the backend itself errors — propagate vs degrade-to-miss              |
| `CacheReadPolicy` | `"strict-mode" \| "cross-mode-readable"` | Per-fingerprint              | Cross-`runtimeMode` boundary — can a `live` turn read a row written by `headless-replay`? |

`strict-fail` is correct for `headless-replay` (a missed hit breaks
determinism). `best-effort` is correct for interactive production
traffic (a cache outage must not break user-facing calls). The two
policies compose — `strict-mode` + `best-effort` is the common
production shape.

## The default `memoryCacheBackend` [#the-default-memorycachebackend]

**The substrate default.** PA-2 C2 Phase 3 promoted
`memoryCacheBackend` from "must be configured" to default-constructed
in the `SessionRuntime` constructor. Every runtime ships with a live
cache out of the box — no `cacheBackend` field on
`SessionRuntimeConfig` is required.

Default cap: **1000 entries / 64 MB**, whichever fills first.
Eviction is LRU over insertion order — `get()` re-inserts on hit,
so the oldest unread entry is the next eviction candidate.

```typescript
import { SessionRuntime } from "@pleach/core";
import { createMemoryCacheBackend } from "@pleach/core/cache";

// Default — runtime constructs a memoryCacheBackend with 1000/64MB caps.
const runtime = new SessionRuntime({
  storage,
  userId: "user_123",
});

// Override at construction with a larger memory cache.
const runtime2 = new SessionRuntime({
  storage,
  userId: "user_123",
  cacheBackend: createMemoryCacheBackend({
    maxEntries: 10_000,
    maxBytes: 256 * 1024 * 1024,
  }),
});

// Opt out — pass `null` to disable caching entirely.
const runtime3 = new SessionRuntime({
  storage,
  userId: "user_123",
  cacheBackend: null,
});
```

A custom backend (Redis, Memcached, Cloudflare KV) implements the
same `CacheBackend` interface and slots into the same field. The
seam doesn't care which it got.

### Reach: all four seam factories [#reach-all-four-seam-factories]

The runtime threads `cacheBackend` through to every seam
construction site. The four seam factories — `synthesizeSeam`,
`reasoningSeam`, `utilitySeam`, `converseSeam` — accept
`cacheBackend?: CacheBackend` on their `Create<Class>SeamOptions`
and forward it to the inner `createSeam(...)`. So the cache is
live at every call class out of the gate; you don't need to wire
it per-seam.

## `prepareCacheInputs` adapter callback [#preparecacheinputs-adapter-callback]

Hosts hook this to canonicalize what enters the cache key.
Without it, volatile fields (request timestamps, monotonic
correlation ids, per-request trace headers) would pollute the
fingerprint and depress the hit rate without changing the
semantic input.

The callback receives the prepared input the seam is about to
fingerprint and returns the canonicalized form. The substrate
ships a no-op default — every input passes through unchanged. See
[`src/cache/`](https://github.com/pleachhq/core/tree/main/src/cache)
for the exact signature; the shape is intentionally narrow so
hosts can express "strip these fields" without subscribing to a
larger preprocessing pipeline.

A typical implementation strips request-scoped metadata before
fingerprinting, leaving the system prompt, the message array,
the tool list, and the model id intact.

## The four fingerprint gaps [#the-four-fingerprint-gaps]

The cache key includes four fields that consumers must be aware
of:

| Field          | Why it's in the key                                                                                                                       |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `systemPrompt` | The substrate's prompt fragments expand per turn; two turns with different system prompts produce different outputs by design.            |
| `temperature`  | Sampling temperature changes the response distribution; reusing a `temperature: 0.2` row for a `temperature: 0.9` request would be wrong. |
| `runtimeMode`  | A turn running in `live` mode reads provider state a `headless-replay` turn doesn't see; the key reflects that.                           |
| `tenantId`     | Per-tenant prompt overrides and tool catalogs differ; cross-tenant reads would leak one tenant's prepared input into another's call.      |

Two reads with different values across any of those four miss the
cache. By design. The substrate trades hit rate for correctness —
a hit that returns the wrong response is worse than a miss that
re-invokes the provider.

## Two read modes [#two-read-modes]

The `CacheReadPolicy` governs how strict the `runtimeMode` gap
is at read time. Default is `strict-mode`.

### `strict-mode` [#strict-mode]

A turn running in mode A cannot read a row written by mode B.
The key includes `runtimeMode` and the lookup fails when the
modes differ.

This is the safe default. `live`-mode turns may have invoked
non-deterministic tools whose results shouldn't replay verbatim
under `headless-replay`; isolating reads by mode keeps each mode's
cache semantics independent.

### `cross-mode-readable` [#cross-mode-readable]

A turn may read a row written under a different mode when the
prepared input is mode-independent. The seam evaluates the
mode-independence claim at read time; the row is returned only
when the claim holds.

Cross-mode reads stay in soak today. Hosts should keep the
default until the cross-mode contract clears — see
[Versioning](/docs/versioning) for how the gate moves.

## Telemetry [#telemetry]

`metricsSnapshot()` returns live counters. Two of the four fields
matter for storage-volume monitoring on production:

| Field             | What it answers                              |
| ----------------- | -------------------------------------------- |
| `hits` / `misses` | Is the cache earning its keep?               |
| `entryCount`      | How many distinct fingerprints have we seen? |
| `sizeBytes`       | Are we approaching the configured cap?       |

```typescript
const snapshot = runtime.getCacheBackend().metricsSnapshot();
console.log(
  `cache: ${snapshot.hits} hits / ${snapshot.misses} misses`,
  `(${snapshot.entryCount} entries, ${snapshot.sizeBytes} bytes)`,
);
```

The CI gate `audit:c2-cache-hit-rate-clean` is the canonical
hit-rate signal in production. It accumulates per-canvas-batch
samples of `[UXParity:c2-cache-hit-rate]` emissions and fails
`:strict` until a 3-batch soak window shows hit-rate ≥ 5% and
median hit-latency \< 50 ms. The clock starts on the first
production canvas batch after the default-promotion deploy; the
ledger lives in-repo at
`scripts/audit/c2-cache-hit-rate-clean.soak-ledger.json`.

Wire the snapshot into your existing metrics pipeline the same way
you wire the audit ledger — see [Observability](/docs/observability)
for the per-call decorator pattern; cache metrics are coarser, so a
periodic snapshot read into a gauge is enough.

## What's not in scope today [#whats-not-in-scope-today]

Cross-mode read of cached state across `runtimeMode` transitions
remains in soak. The contract isn't sealed yet — edge cases around
tool-call replay and async-job resumption are still being walked.

Keep `strict-mode` until the gate clears. The cost of the stricter
read is one extra provider call when a mode boundary is crossed;
the cost of a premature cross-mode hit is a correctness bug that
won't surface until the row replays. See
[Versioning](/docs/versioning) for the gate-moves-the-mode rollout
shape.

The in-memory backend is **single-process**. Each `SessionRuntime`
instance owns its own `Map`; two replicas of the same host process
don't share cache state. For multi-process or multi-replica
deployments — autoscaling Vercel / Cloudflare Workers, fanned-out
Kubernetes pods, anything beyond a single Node process — swap in a
shared backend (Redis, Memcached, Cloudflare KV, Supabase) by
implementing the same `CacheBackend` contract.

The substrate does not ship a Redis or Supabase cache backend
today. Both are straightforward against the contract above — the
in-tree memory backend is the reference implementation and the
shape consumer adapters mirror.

## What not to do [#what-not-to-do]

A few patterns that fight the substrate:

* **Don't substitute the cache for an audit log.** The cache is
  best-effort and evictable; the [audit ledger](/docs/audit-ledger)
  is append-only and durable. A cache entry that aged out is gone;
  an audit row is not. Cost attribution, compliance evidence, and
  replay all read the ledger.
* **Don't put PII into prepared inputs without scrubbing first.**
  The cache key derives from the prepared input; a leaked field
  there shows up in cross-tenant hit-rate diagnostics later. Run
  inputs through [scrubbers](/docs/scrubbers) before the seam
  fingerprints them.
* **Don't construct a `SessionRuntime` with a custom cache that
  throws synchronously.** The seam invokes `set` without awaiting;
  a synchronous throw escapes the fire-and-forget contract and
  breaks the turn. Wrap I/O in `try`/`catch` inside the backend
  and degrade to a counter increment.
* **Don't tune the fingerprint to chase hit rate.** The four gaps
  are load-bearing for correctness. A higher hit rate that comes
  from collapsing `tenantId` or `runtimeMode` is a cross-tenant
  bug waiting to ship.

## Where to go next [#where-to-go-next]

<Cards>
  <Card title="Prompt caching" href="/docs/prompt-caching" description="The provider-side prefix-reuse sibling — Anthropic/OpenAI server-side caching the seam also rides on." />

  <Card title="Scrubbers" href="/docs/scrubbers" description="Run prepared inputs through scrubbers before they reach the cache key." />

  <Card title="SessionRuntime" href="/docs/session-runtime" description="The constructor that holds the `cacheBackend` field." />

  <Card title="Versioning" href="/docs/versioning" description="How the cross-mode-readable gate clears soak and becomes the default." />
</Cards>