Tamper-evident hash chain

The hash chain is one of three concepts in the audit-ledger cluster — the AuditableCall row is what it protects; the ProviderDecisionLedger is the write path; this page covers the tamper-evidence layer that links every persisted row to its predecessor. For read-side observability (OTel spans, lineage, Datadog wiring) see Observability.

The hash chain protects the event log against three after-the-fact tampers: a silent backfill that inserts a row at an earlier index, a reorder that swaps two adjacent rows, and a removal that drops a row from the middle of the stream. Each of those mutations breaks the linkage between a row's prev_hash and the previous row's row_hash, and verification reports the first index where the chain breaks.

The chain doesn't protect against a compromised writer mutating rows in real time — a writer that owns both the row contents and the hash stamp can produce a self-consistent chain on top of any state it likes. Real-time write integrity is a runtime-attestation problem and out of scope for this page.

The two columns

The chain rides on two columns added to harness_event_log:

Both columns store the raw 32-byte digest as BYTEA (not text) — the verifier keeps storage compact and the walk O(rows); it exports hex only for display (encode(row_hash, 'hex')). Both are nullable for back-compat: rows written before the migration landed carry NULL in both, and stamped rows written after carry both. The migration shape:

ALTER TABLE harness_event_log
  ADD COLUMN IF NOT EXISTS prev_hash BYTEA,
  ADD COLUMN IF NOT EXISTS row_hash  BYTEA;

CREATE INDEX IF NOT EXISTS harness_event_log_chat_row_hash_idx
  ON harness_event_log (chat_id, sequence_number DESC)
  INCLUDE (row_hash)
  WHERE chat_id IS NOT NULL AND row_hash IS NOT NULL;

The partial index keys the verification walk: a chat's stamped slice is read in (chat_id, sequence_number) order without a sequential scan over pre-hash rows. One caveat — sequence_number is a per-chat write ordinal, not a uniqueness guarantee. Under concurrent writers a slice can carry a repeated ordinal, so a verifier should treat sequence_number as a hint and resolve ties by a monotonic insertion identity (created_at, id); ordering the walk on sequence_number alone can report false breaks when two rows share an ordinal.

What "canonical encoding" means

Two writers that observed the same row state need to compute the same row_hash, otherwise the chain isn't verifiable across writer restarts or across writers running in parallel. The encoding is a deterministic byte serialization of the row's columns in a fixed order, with JSONB normalized to sorted-key form and timestamps serialized as their ISO-8601 string.

The exact column order and the JSONB normalization rules live in the source — see the linked src/event-log/ directory. Don't reimplement the encoding from a guess; consume the helper the writer uses so a future column addition propagates through both the stamp and the verifier in one place.

Lifecycle status

Be explicit about where the chain sits today.

Today. The prev_hash and row_hash columns ship in the @pleach/core schema bundle (003_harness_event_log.sql), so a bare npx pleach init install has columns to stamp into — no host-only migration required. Writer-side stamping is on by default (EventLogWriter's c9PhaseBEnabled, a single-character rollback if you need to disable it): the writer reads the most recent row_hash for the (tenant_id, chat_id) slice, computes row_hash over the canonical encoding of the row it's about to insert, and writes both columns as part of the insert. Rows written before the columns existed carry NULL in both and are skipped by the verifier. In a reference deployment, every row over a rolling 24-hour window carried both columns.

Concurrency note. The per-chat sequence_number is assigned by reading the current max and incrementing; parallel writers can therefore collide on an ordinal (see the verification-walk caveat above). The chain links themselves are unaffected — prev_hash still copies the prior row_hash verbatim — but a verifier must order by insertion identity, not sequence_number, to walk it.

Shipping. The verifier ships in code on @pleach/core/eventLog: verifyChainForChat(...) walks the chain and reports the first index where it breaks, and generateProof(...) produces a portable ChainProofV1 over a window of the chain. @pleach/replay consumes verifyChainForChat to gate deterministic re-execution against an untampered log. Because the same chainStep, computeRowHash, computeGenesisSeed, canonicalizeRowForChain, and PLEACH_C9_CANONICALIZATION_VERSION helpers the writer uses are exported from @pleach/core/eventLog, verifier and writer share canonical encoding by construction. The SQL pattern below remains the fallback for ad-hoc audits or hosts that haven't wired the in-code verifier.

C9 probes — proof the writer is reaching prod

Two [UXParity:c9-hash-chain-*] probes are wired into EventLogWriter and hashChain.ts. They activate the dormant audit:c9-hash-chain-integrity soak ledger so the 3-batch clean gate can become load-bearing for the verifier-CLI cutover. Both probes fire whether or not stamping is enabled — a missing emission means a misconfigured rollout, not a passing one.

[UXParity:c9-hash-chain-row-stamp] (PE-1)

Fires once per chat-bearing flush row in EventLogWriter.flushBatchWithRetry. Two phases:

• phase: "active" — c9PhaseBEnabled === true, chainStep stamps prev_hash + row_hash; the probe carries the first 16 hex chars of each.
• phase: "disabled" — operator opt-out path; prevHashPrefix and rowHashPrefix are null so canvas-grep cohorts can split active-vs-disabled fire counts without rerunning the cohort.

Payload:

interface C9HashChainRowStampInput {
  phase: "active" | "disabled";
  chatId: string;
  tenantId: string;
  prevHashPrefix: string | null;  // 16 hex chars when active; null when disabled
  rowHashPrefix: string | null;
  // emitted with a tsMs timestamp
}

[UXParity:c9-hash-chain-verify] (PE-2)

Fires once per verifyChainForChat call at the function epilogue — the verifier walks the iterator collecting counters, then emits one structured line with the totals + verdict. Locked at every call, no sampling.

Payload:

interface C9HashChainVerifyInput {
  chatId: string;
  tenantId: string;
  chainValid: boolean;
  rowCount: number;            // total rows examined (legacy + non-legacy)
  nonLegacyRowCount: number;   // chain-participating subset
  failedIndex?: number;        // present iff chainValid: false
  warnOnly: boolean;
}

chainValid: false is the structured JSON complement to the existing [UXParity:c9-chain-verify-warn-only] console.warn.

Audit gate clean condition

The audit:c9-hash-chain-integrity gate evaluates a batch as clean when three conditions hold over a three-batch window:

• c9-hash-chain-row-stamp count > 0 (any write-shadow emission proves the probe is reaching production)
• Every c9-hash-chain-verify emit carries chainValid: true
• Zero [C9:legacy-prefix] boundary-disagreement diagnostics

A single failing batch doesn't fail the deploy — the three-batch aggregation tolerates a transient sink delay. The activation gate flipping green is what promotes the writer from shadow to enabled.

Pure hash module + verification

The chain's hashing and verification logic ships today as a pure module — no substrate imports, no writer wiring, no database access. It builds on top of node:crypto and nothing else, so verifiers, tests, and external auditors can consume the same canonicalization the writer will use without pulling in the rest of the runtime.

import {
  PLEACH_C9_CANONICALIZATION_VERSION,  // "pleach.c9.v1"
  computeGenesisSeed,                  // (tenantId, chatId) → Buffer
  canonicalizeRowForChain,             // (CanonicalRowFields) → Buffer
  computeRowHash,                      // (prevHash, canonical) → Buffer
  chainStep,                           // genesis-aware single advance
  verifyChain,                         // walk + row-precise diagnostic
  isLegacyRow,                         // null-rowHash detector
} from "@pleach/core/eventLog";

The genesis seed is derived per (tenant_id, chat_id) and carries the version prefix into the hash, so a chain rooted at one (tenant, chat) pair can't be grafted onto another. verifyChain walks a slice top-to-bottom and returns either { ok: true } or a row-precise diagnostic — { ok: false, failedIndex, expected, actual, reason } — pinpointing the first index where the stored chain diverges from a recomputed one. isLegacyRow separates pre-stamping rows from chain breaks so the verifier can emit a distinct legacy signal rather than reporting a false tamper.

The next phases — writer-side stamping wiring at insert time, and the @pleach/replay verification CLI surface previewed below — build on this module. The canonicalization contract lives here so both directions read from a single source.

What gets hashed, what doesn't

The chain hashes the persisted shape of the row — the bytes that actually live in harness_event_log. Scrubber-redacted payloads hash to the redacted form, not the pre-redaction form. That's deliberate: the chain protects against tampering with what was written, not against losing information to redaction.

See Scrubbers for the redaction layer that shapes the payload before it reaches the writer.

Back-compat for pre-hash rows

A consumer that started its chain mid-history doesn't need to backfill old rows. Verification skips any row whose row_hash is NULL and resumes at the next stamped row. The first stamped row in a tenant's history is the chain root for that tenant; everything before it is unverified by construction.

In practice: a tenant who upgraded after the stamping flag flipped has a verifiable chain from the upgrade row forward. Older rows remain queryable; they just aren't covered by the chain.

Verification (preview)

When the verification CLI ships in @pleach/replay, the surface will look roughly like:

npx pleach-replay verify-chain \
  --session <session-id> \
  --from <record-id>
# → ok through 12,847 rows
# → break at record_id 01JCXY... (prev_hash mismatch)

Until then, verification is a recursive CTE that walks the chain and reports the first row where prev_hash doesn't match the previous row_hash:

WITH RECURSIVE chain AS (
  SELECT
    record_id,
    prev_hash,
    row_hash,
    1 AS idx
  FROM harness_event_log
  WHERE session_id = $1
    AND row_hash IS NOT NULL
  ORDER BY record_id ASC
  LIMIT 1
),
walked AS (
  SELECT c.record_id, c.prev_hash, c.row_hash, c.idx
  FROM chain c
  UNION ALL
  SELECT
    e.record_id,
    e.prev_hash,
    e.row_hash,
    w.idx + 1
  FROM walked w
  JOIN harness_event_log e
    ON e.session_id = $1
   AND e.record_id > w.record_id
   AND e.row_hash IS NOT NULL
   AND e.prev_hash = w.row_hash
  ORDER BY e.record_id ASC
  LIMIT 1
)
SELECT idx, record_id FROM walked ORDER BY idx DESC LIMIT 1;

The last idx returned is the chain length. If it's shorter than the count of stamped rows for the session, the chain breaks at the next record_id after the returned one. The CLI will do this walk plus a contents re-hash (recomputing row_hash from the row's columns and comparing it to the stored value) — the SQL pattern catches link breaks but not contents tampering on a single row.

Relationship to @pleach/replay

Replay reads the chain to assert deterministic re-execution against an untampered event log. A replay pass that walks a slice whose chain doesn't verify can't claim its result reproduces the original session — the inputs to the replay aren't trusted. The verification step is what lets replay diff results stand as evidence.

See Eval and replay for the replay surface itself.

What the hash chain doesn't replace

The chain is an after-the-fact tamper detector. It's not access control. Specifically, it does not replace:

• RLS. Row-level security gates who can read or write rows. The chain runs over whatever rows actually got written; it doesn't decide who got to write them.
• Auth. A signed JWT proves the requester is who they say they are. The chain says nothing about identity — it says only that the row sequence hasn't been mutated since it was written.
• In-flight encryption. TLS protects bytes between the writer and the database. The chain protects bytes after they land.

Layer the chain on top of those controls, not in place of them.

Where to go next