Tamper-evident hash chain

The hash chain is one of three concepts in the audit-ledger cluster — the AuditableCall row is what it protects; the ProviderDecisionLedger is the write path; this page covers the tamper-evidence layer that links every persisted row to its predecessor. For read-side observability (OTel spans, lineage, Datadog wiring) see Observability.

The hash chain protects the event log against three after-the-fact tampers: a silent backfill that inserts a row at an earlier index, a reorder that swaps two adjacent rows, and a removal that drops a row from the middle of the stream. Each of those mutations breaks the linkage between a row's prev_hash and the previous row's row_hash, and verification reports the first index where the chain breaks.

The chain doesn't protect against a compromised writer mutating rows in real time — a writer that owns both the row contents and the hash stamp can produce a self-consistent chain on top of any state it likes. Real-time write integrity is a runtime-attestation problem and out of scope for this page.

What can bound it is an outside witness. If an independent party has already signed the chain's merkle root, a rewrite of any row under that root stops being undetectable: re-deriving the root from the mutated rows no longer reproduces the value that was witnessed. The forger controls the chain, but not the witness's signature.

That is what periodic RFC 3161 anchoring provides. The distinction that matters is when the witnessing happens: anchoring only at attestation time bounds nothing in between — a writer can rewrite freely and attest afterwards, and the result is self-consistent. Anchoring on a schedule converts "history could have been rewritten at any point" into "history could have been rewritten at most one interval ago." The anchor interval is the bound, so it is the number worth choosing deliberately rather than inheriting.

This does not make the chain tamper-proof, and it should not be described that way. It makes a rewrite of already-witnessed rows tamper-evident to anyone who re-derives the root and checks it against the witnessed one — which means the property is only real if something actually performs that check.

The two columns

The chain rides on two columns added to harness_event_log:

Both columns store the raw 32-byte digest as BYTEA (not text) — the verifier keeps storage compact and the walk O(rows); it exports hex only for display (encode(row_hash, 'hex')). Both are nullable for back-compat: rows written before the migration landed carry NULL in both, and stamped rows written after carry both. The migration shape:

ALTER TABLE harness_event_log
  ADD COLUMN IF NOT EXISTS prev_hash BYTEA,
  ADD COLUMN IF NOT EXISTS row_hash  BYTEA;

CREATE INDEX IF NOT EXISTS harness_event_log_chat_row_hash_idx
  ON harness_event_log (chat_id, sequence_number DESC)
  INCLUDE (row_hash)
  WHERE chat_id IS NOT NULL AND row_hash IS NOT NULL;

The partial index keys the verification walk: a chat's stamped slice is read in (chat_id, sequence_number) order without a sequential scan over pre-hash rows. One caveat — sequence_number is a per-chat write ordinal, not a uniqueness guarantee. Under concurrent writers a slice can carry a repeated ordinal, so a verifier should treat sequence_number as a hint and resolve ties by a monotonic insertion identity (created_at, id); ordering the walk on sequence_number alone can report false breaks when two rows share an ordinal.

What "canonical encoding" means

Two writers that observed the same row state need to compute the same row_hash, otherwise the chain isn't verifiable across writer restarts or across writers running in parallel. The encoding is a deterministic byte serialization of the row's columns in a fixed order, with JSONB normalized to sorted-key form and timestamps serialized as their ISO-8601 string.

The exact column order and the JSONB normalization rules live in the source — see the linked src/event-log/ directory. Don't reimplement the encoding from a guess; consume the helper the writer uses so a future column addition propagates through both the stamp and the verifier in one place.

Lifecycle status

Be explicit about where the chain sits today.

Today. The prev_hash and row_hash columns ship in the @pleach/core schema bundle (003_harness_event_log.sql), so a bare npx pleach init install has columns to stamp into — no host-only migration required. Writer-side stamping is on by default (EventLogWriter's c9PhaseBEnabled, a single-character rollback if you need to disable it): the writer reads the most recent row_hash for the (tenant_id, chat_id) slice, computes row_hash over the canonical encoding of the row it's about to insert, and writes both columns as part of the insert. Rows written before the columns existed carry NULL in both and are skipped by the verifier. In a reference deployment, every row over a rolling 24-hour window carried both columns.

Concurrency note. The per-chat sequence_number is assigned by reading the current max and incrementing; parallel writers can therefore collide on an ordinal (see the verification-walk caveat above). The chain links themselves are unaffected — prev_hash still copies the prior row_hash verbatim — but a verifier must order by insertion identity, not sequence_number, to walk it.

Shipping. The verifier ships in code on @pleach/core/eventLog: verifyChainForChat(...) walks the chain and reports the first index where it breaks, and generateProof(...) produces a portable ChainProofV1 over a window of the chain. @pleach/replay consumes verifyChainForChat to gate deterministic re-execution against an untampered log. Because the same chainStep, computeRowHash, computeGenesisSeed, canonicalizeRowForChain, and PLEACH_C9_CANONICALIZATION_VERSION helpers the writer uses are exported from @pleach/core/eventLog, verifier and writer share canonical encoding by construction. The SQL pattern below remains the fallback for ad-hoc audits or hosts that haven't wired the in-code verifier.

C9 probes — proof the writer is reaching prod

Two [UXParity:c9-hash-chain-*] probes are wired into EventLogWriter and hashChain.ts. They activate the dormant audit:c9-hash-chain-integrity soak ledger so the 3-batch clean gate can become load-bearing for the verifier-CLI cutover. Both probes fire whether or not stamping is enabled — a missing emission means a misconfigured rollout, not a passing one.

[UXParity:c9-hash-chain-row-stamp] (PE-1)

Fires once per chat-bearing flush row in EventLogWriter.flushBatchWithRetry. Two phases:

• phase: "active" — c9PhaseBEnabled === true, chainStep stamps prev_hash + row_hash; the probe carries the first 16 hex chars of each.
• phase: "disabled" — operator opt-out path; prevHashPrefix and rowHashPrefix are null so canvas-grep cohorts can split active-vs-disabled fire counts without rerunning the cohort.

Payload:

interface C9HashChainRowStampInput {
  phase: "active" | "disabled";
  chatId: string;
  tenantId: string;
  prevHashPrefix: string | null;  // 16 hex chars when active; null when disabled
  rowHashPrefix: string | null;
  // emitted with a tsMs timestamp
}

[UXParity:c9-hash-chain-verify] (PE-2)

Fires once per verifyChainForChat call at the function epilogue — the verifier walks the iterator collecting counters, then emits one structured line with the totals + verdict. Locked at every call, no sampling.

Payload:

interface C9HashChainVerifyInput {
  chatId: string;
  tenantId: string;
  chainValid: boolean;
  rowCount: number;            // total rows examined (legacy + non-legacy)
  nonLegacyRowCount: number;   // chain-participating subset
  failedIndex?: number;        // present iff chainValid: false
  warnOnly: boolean;
}

chainValid: false is the structured JSON complement to the existing [UXParity:c9-chain-verify-warn-only] console.warn.

Audit gate clean condition

The audit:c9-hash-chain-integrity gate evaluates a batch as clean when three conditions hold over a three-batch window:

• c9-hash-chain-row-stamp count > 0 (any write-shadow emission proves the probe is reaching production)
• Every c9-hash-chain-verify emit carries chainValid: true
• Zero [C9:legacy-prefix] boundary-disagreement diagnostics

A single failing batch doesn't fail the deploy — the three-batch aggregation tolerates a transient sink delay. The activation gate flipping green is what promotes the writer from shadow to enabled.

Pure hash module + verification

The chain's hashing and verification logic ships today as a pure module — no substrate imports, no writer wiring, no database access. It builds on top of node:crypto and nothing else, so verifiers, tests, and external auditors can consume the same canonicalization the writer will use without pulling in the rest of the runtime.

import {
  PLEACH_C9_CANONICALIZATION_VERSION,  // "pleach.c9.v1"
  computeGenesisSeed,                  // (tenantId, chatId) → Buffer
  canonicalizeRowForChain,             // (CanonicalRowFields) → Buffer
  computeRowHash,                      // (prevHash, canonical) → Buffer
  chainStep,                           // genesis-aware single advance
  verifyChain,                         // walk + row-precise diagnostic
  isLegacyRow,                         // null-rowHash detector
} from "@pleach/core/eventLog";

The genesis seed is derived per (tenant_id, chat_id) and carries the version prefix into the hash, so a chain rooted at one (tenant, chat) pair can't be grafted onto another. verifyChain walks a slice top-to-bottom and returns either { ok: true } or a row-precise diagnostic — { ok: false, failedIndex, expected, actual, reason } — pinpointing the first index where the stored chain diverges from a recomputed one. isLegacyRow separates pre-stamping rows from chain breaks so the verifier can emit a distinct legacy signal rather than reporting a false tamper.

The next phases — writer-side stamping wiring at insert time, and the @pleach/replay verification CLI surface previewed below — build on this module. The canonicalization contract lives here so both directions read from a single source.

What gets hashed, what doesn't

The chain hashes the persisted shape of the row — the bytes that actually live in harness_event_log. Scrubber-redacted payloads hash to the redacted form, not the pre-redaction form. That's deliberate: the chain protects against tampering with what was written, not against losing information to redaction.

See Scrubbers for the redaction layer that shapes the payload before it reaches the writer.

Back-compat for pre-hash rows

A consumer that started its chain mid-history doesn't need to backfill old rows. Verification skips any row whose row_hash is NULL and resumes at the next stamped row. The first stamped row in a tenant's history is the chain root for that tenant; everything before it is unverified by construction.

In practice: a tenant who upgraded after the stamping flag flipped has a verifiable chain from the upgrade row forward. Older rows remain queryable; they just aren't covered by the chain.

Verification (preview)

When the verification CLI ships in @pleach/replay, the surface will look roughly like:

npx pleach-replay verify-chain \
  --session <session-id> \
  --from <record-id>
# → ok through 12,847 rows
# → break at record_id 01JCXY... (prev_hash mismatch)

Until then, verification is a recursive CTE that walks the chain and reports the first row where prev_hash doesn't match the previous row_hash:

WITH RECURSIVE chain AS (
  SELECT
    record_id,
    prev_hash,
    row_hash,
    1 AS idx
  FROM harness_event_log
  WHERE session_id = $1
    AND row_hash IS NOT NULL
  ORDER BY record_id ASC
  LIMIT 1
),
walked AS (
  SELECT c.record_id, c.prev_hash, c.row_hash, c.idx
  FROM chain c
  UNION ALL
  SELECT
    e.record_id,
    e.prev_hash,
    e.row_hash,
    w.idx + 1
  FROM walked w
  JOIN harness_event_log e
    ON e.session_id = $1
   AND e.record_id > w.record_id
   AND e.row_hash IS NOT NULL
   AND e.prev_hash = w.row_hash
  ORDER BY e.record_id ASC
  LIMIT 1
)
SELECT idx, record_id FROM walked ORDER BY idx DESC LIMIT 1;

The last idx returned is the chain length. If it's shorter than the count of stamped rows for the session, the chain breaks at the next record_id after the returned one. The CLI will do this walk plus a contents re-hash (recomputing row_hash from the row's columns and comparing it to the stored value) — the SQL pattern catches link breaks but not contents tampering on a single row.

Verifying the anchors

Chain verification and anchor verification answer different questions, and the second is not implied by the first. Walking prev_hash proves the rows you can see form an unbroken sequence. It says nothing about whether that sequence is the one an outside witness actually saw.

The gap is concrete rather than theoretical. Delete a row from the middle of a chain and re-link the rows after it onto the row before, and the link walk above accepts the result — every prev_hash matches its predecessor, because you fixed them all. Re-deriving the merkle root does not accept it: the root no longer reproduces the value that was witnessed. A chain that verifies is not the same thing as a chain that matches its witness, and only the second survives a forger who controls the writer.

verifyChainAnchors does that re-derivation. It rebuilds each witnessed segment's root from the live rows and compares it against the root stored on the anchor:

import { verifyChainAnchors } from "@pleach/core/eventLog";

const report = verifyChainAnchors({
  anchors,        // the stored anchor rows for this scope
  entries,        // the live chained rows they claim to witness
  verifyResult,   // the result of verifying the chain itself
  tenantId,
  chatId,
});

Verdicts are ordered by severity, and the ordering is deliberate: root-mismatch is the loudest signal the system can produce, and chain-unverifiable is reported distinctly rather than folded into it — if the chain itself doesn't verify, nothing can be concluded about whether its root matches, and saying otherwise would manufacture a tamper alarm out of a broken input.

An anchor nobody checks is worse than none

The interval is the bound, so the bound is only real while anchors keep arriving. A table that stopped growing an hour ago still has rows in it; every one is still a valid witness of a past root, and not one of them bounds the present. Grade liveness on staleness, never on row count.

import { evaluateChainAnchorFreshness } from "@pleach/core/eventLog";

const liveness = evaluateChainAnchorFreshness({
  anchors,
  intervalMs,     // your anchoring cadence — core owns no schedule
});
// → { status: "fresh" | "stale" | "never-anchored",
//     anchorCount, timestampedAnchorCount, ageMs, thresholdMs, detail? }

anchorCount and timestampedAnchorCount are separate on purpose. A report can hold a hundred rows of which one carries a readable timestamp, and freshness is graded only on the readable ones — so the counts, not the status, are what tell you the stored evidence is intact. Damage to the rows and staleness of the newest one are independent failures and a consumer should read both.

intervalMs is validated rather than trusted: a non-finite or non-positive value throws, because it is the input the entire service level derives from and every degraded reading it could produce would be quieter than the truth.

Severity, and two rules worth copying

classifyChainAnchorAlertLevel turns a liveness report into "ok" | "info" | "warn". Two decisions in it generalise beyond this package:

Liveness never shares a channel with tampering. There is no "alarm" level here. A root mismatch means someone rewrote history; a stale anchor means only that the bound cannot currently be proven, which has mundane causes — an authority blip, a cold start, an operator who switched anchoring on a minute ago. Routing the second onto the first puts frequent benign events on the one signal that must never be ignored, and a signal that cries wolf gets muted.

Damage raises a verdict and can never lower one. Corruption of the stored anchors is combined with the liveness verdict by taking the louder of the two, not by replacing it. The obvious implementation — check corruption first, return early — means one unreadable row turns a warn into an info, so a writer bug that stops stamping timestamps would quieten the alert about the lapsed bound it caused.

Both are instances of one rule: missing or damaged evidence may withhold an escalation, but it must never produce a de-escalation.

Running it on a schedule

The primitives above are per-chat and unopinionated about when they run. @pleach/recipes composes them into the scheduled pass — discover, anchor, persist, verify — as chainAnchorScheduler. It exposes runOnce() and no loop, so the schedule stays yours.

What is not claimed

Tier A — re-deriving the root and comparing it to the stored value — needs no trust anchors and always runs. It proves the chain still hashes to the value you stored. It does not verify the timestamp token itself; that needs pinned CA roots and lives in @pleach/compliance. A skipped token check is reported as skipped and is never a pass.

Keyed chains (HMAC v2)

The v1 chain hashes rows with unkeyed SHA-256, so it proves linkage — but an actor with database write access can mutate a row and recompute every hash after it, producing a self-consistent chain. The scheduled anchoring above is what bounds that actor.

The keyed mode raises the bar at the row level. Pass a signing key as EventLogWriter's c9SigningKey and new rows are chained with HMAC-SHA256 row_hashes under that secret — recompute-resistant: an actor with write access but without the key cannot forge a matching row_hash, so a database-level rewrite breaks verification instead of surviving it. Each keyed row records chain_key_version = 'pleach.c9.v2', so the verifier resolves the mode per chat; omit the key and the chain stays byte-identical v1. The mode requires the chain_key_version column migration.

The threat it closes is the database-level rewriter. A compromised writer that holds the key can still restamp — bounding that remains the anchors' job.

Chain segment anchors

A chain whose earliest chained row hangs off neither its genesis seed nor a segment seed can't be walked from its beginning — the verifier fails at index 0, and no read-side logic can manufacture a boundary the ledger never recorded. Only the writer can, by minting one.

With c9SegmentRestartOnBrokenChain enabled, the first block the writer stamps for such a chat is anchored on computeSegmentSeed(tenantId, chatId, blockStart, key), and its first row carries a _chain_segment_anchor payload marker recording the restart reason and the predecessor the writer refused to consume. The marker is stamped before payload_hash and row_hash are computed, so both hashes cover it — it can't be added to or stripped from a row after the fact without breaking verification.

The flag is off by default because minting an anchor is a semantic act on an append-only ledger: it declares the history before that point not vouched for. That's an operator decision, never an upgrade surprise — with the flag off nothing changes, not even the detection query. Restarts are bounded to one per (tenant, chat) per writer instance.

computeSegmentSeed and isSegmentAnchorRow export from @pleach/core/eventLog. On a keyed chain isSegmentAnchorRow needs the signing key to decode a restart; the payload marker stays readable without it, which is why both exist.

Which merkle construction built the root

Two constructions ship, and a proof now says which one it was built under. ChainProofV1 carries merkleConstruction: "legacy" | "rfc6962", and generateProof(...) takes a matching construction option that defaults to "legacy".

The default is deliberate and is not an endorsement. Switching a chat to "rfc6962" changes every root it produces, and existing anchors countersigned the legacy roots — so the choice belongs to a per-chat rollout gate, not to a function default. Defaulting the other way would silently orphan every anchored chat.

"rfc6962" (RFC 6962 §2.1) fixes two demonstrated defects that a standard construction avoids by design:

merkleConstruction is absent on every proof minted before this landed, which is why verifyInclusionProof treats absence as "legacy". Omission is a measurement that the proof predates the field, not a default anyone chose — never infer the construction from anything else on the proof.

Relationship to @pleach/replay

Replay reads the chain to assert deterministic re-execution against an untampered event log. A replay pass that walks a slice whose chain doesn't verify can't claim its result reproduces the original session — the inputs to the replay aren't trusted. The verification step is what lets replay diff results stand as evidence.

See Eval and replay for the replay surface itself.

What the hash chain doesn't replace

The chain is an after-the-fact tamper detector. It's not access control. Specifically, it does not replace:

• RLS. Row-level security gates who can read or write rows. The chain runs over whatever rows actually got written; it doesn't decide who got to write them.
• Auth. A signed JWT proves the requester is who they say they are. The chain says nothing about identity — it says only that the row sequence hasn't been mutated since it was written.
• In-flight encryption. TLS protects bytes between the writer and the database. The chain protects bytes after they land.

Layer the chain on top of those controls, not in place of them.

Where to go next