# Internal knowledge agent (/docs/internal-knowledge-agent) An internal knowledge agent is the use case that exposes the hidden cost of un-attributed answers. A model that's "pretty sure" is a model that gets cited by an employee in a Slack thread, which becomes a process document, which becomes wrong. Pleach's answer: every chunk the retrieval tool returned lives in the [audit row](/docs/auditable-call-row) for that turn. Every answer is replayable against the same chunks. And a safety policy can refuse to answer when no chunk crossed the confidence threshold. **Related shapes.** [Regulated-domain agent](/docs/regulated-domain-agent) if the corpus contains PHI, PII, or other regulated content. [Multi-tenant SaaS agent](/docs/multi-tenant-saas-agent) if one runtime indexes per-tenant corpora. [Customer support agent](/docs/customer-support-agent) if the retrieved answer feeds a multi-turn support thread. ## What you're building [#what-youre-building] An agent that answers questions about your internal documentation — onboarding wikis, runbooks, design docs. It: * Retrieves relevant chunks from a vector store. * Cites each claim with a chunk id and document URL. * Refuses to answer when retrieval confidence is below threshold. The retrieval tool's full input and output land in the [audit ledger](/docs/audit-ledger). The answer is reproducible under [determinism](/docs/determinism): same question + same chunks * same model + same seed = same answer. ## The retrieval tool [#the-retrieval-tool] One [tool](/docs/tools) that the agent calls before composing any answer. The output schema makes provenance non-optional — the model can't return an answer without surfacing the chunks it used. ```typescript // lib/tools/searchInternalDocs.ts import { defineTool } from "@pleach/core"; import { z } from "zod"; const Chunk = z.object({ chunkId: z.string(), docUrl: z.string().url(), docTitle: z.string(), text: z.string(), score: z.number().min(0).max(1), }); export const searchInternalDocs = defineTool({ name: "search_internal_docs", description: "Retrieve up to 8 chunks from internal documentation. Always call this before composing an answer.", input: z.object({ query: z.string().min(4), topK: z.number().int().min(1).max(8).default(5), }), output: z.object({ chunks: z.array(Chunk), maxScore: z.number().min(0).max(1), queryEmbedded: z.array(z.number()), }), async handler({ query, topK }) { const embedding = await embed(query); // Tenant isolation is enforced at the storage/RLS layer (the runtime's // `tenantId` scope) — it is NOT threaded through the tool context. const chunks = await vectorStore.search(embedding, { topK }); return { chunks, maxScore: Math.max(0, ...chunks.map(c => c.score)), queryEmbedded: embedding, }; }, }); ``` `queryEmbedded` is on the output deliberately — it lands in the audit row, so a later reproduction can use the exact embedding vector instead of re-embedding (which would drift if the embedding model is updated). ## The "no chunk, no answer" safety policy [#the-no-chunk-no-answer-safety-policy] A safety policy that gates the final synthesis on retrieval confidence. If the top chunk's score is below threshold, the runtime forces the model to return the standard "I don't know" template instead of synthesizing. ```typescript // lib/safety/noChunkNoAnswer.ts import { defineSafetyPolicy, safetyPolicyId } from "@pleach/core/safety"; export const noChunkNoAnswer = defineSafetyPolicy({ id: safetyPolicyId("knowledge-agent.no-chunk-no-answer"), version: "1.0.0", enforcement: "refusal", scope: { callClass: "synthesize" }, content: ` [Retrieval confidence policy] If the retrieval tool returns no chunks with a score >= 0.6, do not synthesize an answer. Reply with the standard template: "I don't have enough confidence in the internal documentation to answer this. Try rephrasing, or ask a human." Cite the empty-result set as the reason. `.trim(), }); ``` The policy is **capability-subtracting**: it surfaces the operator's stated refusal posture (composed LAST into the system prompt) and lands on the audit row by `id` + `version` so a later review can ask "which turns ran under this rule". See [Safety](/docs/safety) for the contribution shape. ## Runtime construction [#runtime-construction] ```typescript // lib/runtime.ts import { SessionRuntime, AiSdkProvider, definePleachPlugin, appendPrompt } from "@pleach/core"; import { SupabaseAdapter } from "@pleach/core/sessions"; import { SupabaseSaver } from "@pleach/core/checkpointing"; import { createOpenRouter } from "@openrouter/ai-sdk-provider"; const openrouter = createOpenRouter({ apiKey: process.env.OPENROUTER_API_KEY! }); const KNOWLEDGE_SYSTEM_PROMPT = `You answer questions about internal documentation. Rules: - Always call search_internal_docs before answering. - Cite each claim with the chunk_id and doc URL it came from. - If the tool returns no chunks above 0.6 score, say you don't know.`; export function buildKnowledgeRuntime(req: AuthedRequest) { return new SessionRuntime({ provider: new AiSdkProvider({ model: openrouter("anthropic/claude-sonnet-4-5"), maxSteps: 5, }), storage: new SupabaseAdapter({ client: supabase }), checkpointer: new SupabaseSaver({ client: supabase }), plugins: [definePleachPlugin("knowledge-tools", { tools: [searchInternalDocs], safetyPolicies: [noChunkNoAnswer, piiRedaction], prompts: [appendPrompt("knowledge-agent.system", KNOWLEDGE_SYSTEM_PROMPT)], })], tenantId: req.tenantId, userId: req.userId, }); } ``` ## What the ledger sees [#what-the-ledger-sees] A single turn writes (at minimum) three audit rows: | Row | call\_kind | What it carries | | --- | ---------- | ------------------------------------------------------------------------------------------ | | 1 | `llm` | The initial planning call — system prompt, user message, model response with the tool call | | 2 | `tool` | `search_internal_docs` input + full output (chunks, scores, embedding) | | 3 | `llm` | The synthesis call — input now includes the chunks; output is the cited answer | Reproducing the answer is row 2's output replayed into row 3's input. The [`runtimeMode: "replay"`](/docs/eval-and-replay) constructor does this end-to-end. ## Provenance query [#provenance-query] "Show me every answer this week that cited doc X." ```sql select turn_id, user_id, created_at, payload->'output'->'finalText' as answer from harness_auditable_calls where call_kind = 'llm' and created_at >= now() - interval '7 days' and exists ( select 1 from harness_auditable_calls t where t.turn_id = harness_auditable_calls.turn_id and t.call_kind = 'tool' and t.tool_name = 'search_internal_docs' and t.payload->'output'->'chunks' @> jsonb_build_array( jsonb_build_object('docUrl', $1) ) ); ``` If doc X was wrong and got rewritten, this query is the list of answers to revisit. ## Eval: lock the chunks, vary the prompt [#eval-lock-the-chunks-vary-the-prompt] The retrieval output is recorded. Build a fresh [session runtime](/docs/session-runtime) with a new system prompt, replay the recorded turn — the diff tells you whether the prompt change improved synthesis without re-running embedding. ```typescript import { createReplayRuntime } from "@pleach/replay"; const NEW_SYSTEM_PROMPT = `…revised instructions…`; const challenger = new SessionRuntime({ provider: new AiSdkProvider({ model: openrouter("anthropic/claude-sonnet-4-5"), maxSteps: 5, }), storage: new SupabaseAdapter({ client: supabase }), plugins: [definePleachPlugin("knowledge-tools", { tools: [searchInternalDocs], safetyPolicies: [noChunkNoAnswer, piiRedaction], prompts: [appendPrompt("knowledge-agent.system", NEW_SYSTEM_PROMPT)], })], }); const replayRuntime = createReplayRuntime({ sessionRuntime: challenger, tenantId: req.tenantId, }); const replay = await replayRuntime.replayTurn({ chatId: sessionId, tenantId: req.tenantId, messageId: goldenTurnId, }); // `replay.state` is the reconstructed HydratedHarnessState (typed `unknown`): // the re-synthesized answer plus the tool calls, whose chunks came from the ledger. const state = replay.state; console.log(state); ``` ## Project layout [#project-layout] Three adds on top of the [baseline](/docs/project-layout#a-layout-that-works): an `index/` module the retrieval tool calls into, a `safety/` directory for the "no chunk, no answer" rule, and a SQL file for the provenance query QA reads. ``` my-app/ src/ pleach/ runtime.ts # SessionRuntime + tools + safety + storage tools/ search-internal-docs.ts # defineTool — returns chunks with chunkId + sourceUri index/ client.ts # the vector / lexical index the tool calls into ingest.ts # offline corpus ingestion (separate process) safety/ no-chunk-no-answer.ts # defineSafetyPolicy — refuses synthesis without chunk evidence pii-redaction.ts # → /docs/scrubbers app/ api/agents/[id]/route.ts qa/ provenance.sql # "which chunks did turn T cite?" ``` What changes from the baseline: * **`index/` is separate from `tools/`.** The tool is a thin wrapper that turns a query into a chunk list; the index is the store. They have different release cadences — the index gets re-ingested on a corpus refresh schedule; the tool only changes when the chunk shape changes. * **`index/ingest.ts` is a separate process.** Ingestion is long-running and offline; it doesn't share lifetime with the agent runtime. Keep it in `src/pleach/index/` so the chunk shape stays consistent between writer and reader, but run it via its own entry point (cron, queue worker, CLI). * **`safety/no-chunk-no-answer.ts` is the load-bearing file.** This is what prevents [fabricated citations](/docs/fabrication-detection) — a capability-subtracting [safety policy](/docs/safety) that refuses synthesis if the tool returned zero chunks. A prompt instruction would not be enforceable; a policy is. * **`qa/provenance.sql` lives in the repo.** The [provenance query](#provenance-query) is what answers "what did the agent cite?" weeks after the turn. Same discipline as the customer-support rollup: the SQL ships next to the code that produces the rows it reads. ## Where to go next [#where-to-go-next]