AI Agent Incident Postmortem Template: What to Record After a Failure

Traditional incident practice supplies the durable foundation: a blameless account of impact, timeline, trigger, root cause, recovery, lessons, and concrete actions with owners. Google SRE guidance also emphasizes specific data, source links, a single accountable postmortem owner, and action items with verifiable completion criteria.

Agent systems add a second evidence plane. The same visible answer can depend on the active system prompt, model and parameters, tool definitions, retrieved documents, conversation history, routing path, policy version, and external side effects. If those identities are missing, the postmortem may describe the symptom without preserving the system that produced it.

Create one durable incident record and attach evidence rather than reconstructing it from memory. The postmortem owner should leave every unknown explicit. If a required measurement was not collected, write UNOBSERVED instead of presenting a zero or estimate as an observed measurement. If an impact estimate is operationally useful, label its method, range, and uncertainty separately.

Name one representative failed execution and capture its version lineage before deploying a fix. If the incident affected several distinct paths, preserve one case per path instead of combining them into a fictional average trajectory. The evidence should make it possible to identify what the agent saw, decided, called, changed, and returned.

Agent traces can contain prompts, user data, retrieved documents, and tool inputs or outputs. OpenAI's Agents SDK documentation explicitly warns that generation and function spans may capture sensitive data. Follow the organization's privacy, access, and retention policy; preserve bounded references, hashes, or redacted evidence when full content is unnecessary.

A model update, prompt promotion, index refresh, traffic shift, tool outage, or permission change may trigger the incident window. The first causal step is the earliest supported point where behavior diverged. The user-visible symptom is what happened later. These can be three different facts.

Do not assign blame from timing or from the last agent that produced an answer. Preserve the causal edge connecting an earlier decision to the later effect. If the trace does not identify a unique owner or fix type, leave it unresolved and record the exact evidence needed next.

If an LLM judge or heuristic detected the incident, record its prompt or rule version, reference labels, threshold, and known error modes. Preserve the human or deterministic evidence used to confirm the failure. A detector can be useful without being authoritative about causality, severity, or the correct fix.

Do not rewrite the incident to match the judge. Record consequential disagreement and its likely false-positive or false-negative effect. If independent evidence cannot resolve it, mark detection or causality unresolved rather than averaging conflicting judgments into a reassuring score.

Disabling a tool, routing traffic away, or rolling back a prompt can reduce harm without identifying the cause. A candidate can pass replay and regression tests without reaching production. A deployment marker proves what became active, not that the change improved the targeted behavior.

A vague instruction to improve monitoring is not an action item. An alert when a required handoff receipt is missing for five minutes, with an integration test that exercises the timeout path, has an owner and a verifiable end state. Write actions so a reviewer can determine whether they were completed without trusting a status label.

Cover prevention, mitigation, detection, and learning in proportion to the incident. Name the owning system, exact deliverable, due date, completion evidence, and the person responsible for closing the item. A postmortem without subsequent action is only a record of what the team may repeat.

Close incident response when impact is contained, the durable record has one accountable owner, and follow-up work is tracked. Close the corrective-change claim only when its evidence state is explicit. If comparable post-deployment evidence has not been collected, the fix remains UNOBSERVED even when the postmortem itself is complete.

Converra uses the production verdicts verified, not fixed, and confounded for deployed changes. UNOBSERVED is the pre-verdict evidence state in this template, not a fourth Converra production verdict. It means the required production measurement does not yet exist.

Converra connects a targeted production failure to trace-backed diagnosis, a supported corrective candidate, paired simulation, regression checks, governed delivery, and a post-deployment verdict. That lineage can populate the behavioral portions of an incident record and keep a tested change attached to the production result.

The customer still owns incident severity, outcome definitions, privacy and retention policy, service recovery, deployment authority, code or data changes outside supported scope, and final accountability. A completed postmortem, green dashboard, merged pull request, or simulation pass does not by itself prove the behavioral fix worked in production.