# Prompt caching (/docs/prompt-caching)
A cache hit is determined entirely by the fingerprint. The variable
surface the backend stores against it is the recorded response —
`content`, `toolCalls`, `finishReason`, token `usage`, `modelId`,
and (for streaming calls) the chunk sequence — plus
`recordedAt`/`recordedBy` provenance and a `sizeBytes` accounting
field the backend maintains on `set`.
The seam derives the fingerprint key from `@pleach/core/fingerprint`;
this module is only about storage and retrieval semantics.
## Public exports [#public-exports]
| Export | Kind | Purpose |
| --------------------------- | --------- | ----------------------------------------------------- |
| `CacheBackend` | interface | Storage contract every backend implements. |
| `CacheEntry` | type | Recorded response payload + provenance. |
| `CacheGetMode` | union | Per-call failure-mode policy passed to `get()`. |
| `CacheGetOptions` | type | `{ mode: CacheGetMode }` — `get()`'s second argument. |
| `CacheStats` | type | O(1) counters returned by `stats()`. |
| `MemoryCacheBackendOptions` | type | Memory backend constructor options. |
| `createMemoryCacheBackend` | factory | Returns a `CacheBackend` backed by an in-memory LRU. |
## `CacheBackend` [#cachebackend]
Memory and Supabase backends both implement this surface. The seam
holds only this contract.
```typescript
interface CacheBackend {
readonly id: string;
get(key: Fingerprint, options: CacheGetOptions): Promise;
set(key: Fingerprint, entry: CacheEntry): Promise;
delete(key: Fingerprint): Promise;
list(prefix: Partial): AsyncIterable;
stats(): Promise;
}
```
Four invariants every implementation maintains:
1. `get()` returns a deep-frozen entry — callers must not mutate.
2. `set()` is idempotent on fingerprint (last-write-wins).
3. `stats()` is O(1) — counters update on every mutation.
4. `list()` is async-iterable even for memory implementations so
callers handle pagination from day one.
## `CacheGetMode` [#cachegetmode]
The per-call failure-mode policy. Distinct from
`CacheReadPolicy` in `@pleach/core/fingerprint`, which governs
cross-`RuntimeMode` boundary reads.
| Mode | Behavior on backend error | When to use |
| ------------- | ------------------------------- | --------------------------------------------------------------------------------- |
| `strict-fail` | Error propagates to the caller. | `headless-replay` — a missed hit breaks determinism. |
| `best-effort` | Error returns `null` and logs. | Interactive production traffic — a cache outage must not break user-facing calls. |
Same backend instance serves both — the mode is a per-call decision,
not a backend configuration.
## `CacheEntry` [#cacheentry]
The recorded payload. Opaque to the backend; the seam owns shape
validation.
| Field | Type | Notes |
| ----------------------- | ------------------------------------ | --------------------------------------------------------------------------- |
| `fingerprint` | `Fingerprint` | Cache key, replicated onto the entry. |
| `metadata` | `FingerprintMetadata` | Key/metadata split sibling — diagnostic only. |
| `response.content` | string | Final aggregated response text. |
| `response.toolCalls` | `ReadonlyArray` | Provider-shaped tool calls. |
| `response.finishReason` | string | Unmodified provider string. |
| `response.usage` | `{ inputTokens; outputTokens }` | Token accounting. |
| `response.modelId` | string | The model that produced the entry. |
| `streamChunks` | `ReadonlyArray` (optional) | Set on streaming invocations so replay can re-emit deltas in arrival order. |
| `recordedAt` | ISO-8601 | When the entry was written. |
| `recordedBy` | `{ pleachVersion; nodeId; stageId }` | Provenance for cross-version replays. |
| `sizeBytes` | number | Maintained by the backend on `set`; drives LRU eviction. |
## `CacheStats` [#cachestats]
```typescript
interface CacheStats {
readonly id: string;
readonly entryCount: number;
readonly sizeBytes: number;
readonly hits: number;
readonly misses: number;
readonly hitRatio: number;
readonly evictions: number;
}
```
`hitRatio` is `hits / (hits + misses)`, or `0` when both are zero.
Counters update on every mutation, so `stats()` is O(1).
## Memory backend [#memory-backend]
```typescript
import { createMemoryCacheBackend } from "@pleach/core/cache";
const cache = createMemoryCacheBackend({
maxEntries: 1000, // default 1000
maxBytes: 64 * 1024 * 1024, // default 64 MB
id: "memory", // default "memory"
});
const entry = await cache.get(fingerprint, { mode: "best-effort" });
if (!entry) {
const response = await seam.invoke(...);
await cache.set(fingerprint, buildEntry(response));
}
```
The implementation rides on JavaScript's insertion-ordered `Map`:
`get()` re-inserts on hit so the iterator's first entry is the LRU
candidate. Eviction runs after every `set()` until both `maxEntries`
and `maxBytes` caps are satisfied.
## Wiring into a runtime [#wiring-into-a-runtime]
`SessionRuntimeConfig.cacheBackend` is the field every seam reads
through. Pass the backend at construction and every provider call
shares the same cache instance.
```typescript
import { SessionRuntime } from "@pleach/core";
import { createMemoryCacheBackend } from "@pleach/core/cache";
const cacheBackend = createMemoryCacheBackend({ maxEntries: 5_000 });
const runtime = new SessionRuntime({
storage,
userId: "user_123",
cacheBackend,
});
// Later, inspect counters:
const stats = await cacheBackend.stats();
console.log(stats.hits, stats.misses, stats.hitRatio);
```
## Picking a mode per call [#picking-a-mode-per-call]
Same backend instance; different `mode` per `get()`. Production
traffic reads `best-effort` so a backend outage degrades to a miss;
replay reads `strict-fail` so a missed hit fails fast instead of
re-invoking the provider.
```typescript
// Interactive request — a cache outage must not break the user-facing call.
const cached = await cacheBackend.get(fingerprint, { mode: "best-effort" });
if (cached) return cached;
return await seam.invoke(...);
// Replay run — a missed hit means the cache is wrong, not the model.
const replayed = await cacheBackend.get(fingerprint, { mode: "strict-fail" });
if (!replayed) throw new Error(`replay miss: ${fingerprint.promptHash}`);
return replayed;
```
## Where to go next [#where-to-go-next]