# How we write the first draft of every incident postmortem

Once an incident is resolved, a Machine agent reads the incident channel, lines the timeline up with deploys and log spikes, and opens a postmortem draft as a doc PR for the team to review and edit.

Canonical page: https://themachine.myapps.ai/use-cases/incident-postmortem

Everyone agrees postmortems matter, and everyone is tempted to skip them. When
the fire is out and the adrenaline drops, somebody still has to scroll back
through the incident channel, match it against deploys and log spikes, figure
out what really caused it and write the whole thing down. That's an hour of
fiddly work at exactly the moment the team wants to get back to everything
else. So the writeup ends up as three lines, or never happens.

We hand the first draft to an agent on Machine. As soon as an incident is
marked resolved, it reads the incident channel, matches events against deploys
and log spikes, and opens a structured postmortem as a doc PR. The agent writes
the draft; people review it and sign it off. We use this for our own incidents.

- **Team:** Machine
- **Runs on:** An incident is marked resolved and the draft begins
- **Connected systems:** Incident channel · Logs · GitHub
- **Mode:** Opens a doc PR; people make it final

## The problem

What happened during an incident is split across places that don't naturally
match up: the chatter in the incident channel, the deploy history and the log
spikes. Building a timeline means reading the whole channel, pairing each
moment with what was released and what the logs showed, and working out where
the cause most likely lies.

The shortcuts people take are weak. Someone opens an empty template and fills
it in from memory, so the timeline wanders and details get blurred. Or the
incident closes with no writeup, and when the same failure comes back there's
nothing to refer to. Everything needed for a proper postmortem is already
there. Stitching it together by hand is the job nobody wants straight after an
incident.

## What we built

On Machine, closing out an incident sets off an agent. It reads the incident
channel to build the timeline, fetches deploys and log spikes from the same
stretch of time, connects the dots, and writes a structured postmortem with a
timeline, root cause and action items. It submits that as a doc PR to the repo.
The team reviews and edits it like any other change. The agent only ever
writes the draft; it never publishes a finished postmortem itself.

## How it works

### Start when the incident closes

We connect the incident channel to Machine as a **channel**, so marking an
incident resolved is what sets things off. That opens a fresh **session**, a
cloud computer (sandbox) just for this run, loaded with the incident. Each
incident gets one session on one throwaway machine, and the draft gets going
while everyone still remembers the details.

### Teach it our postmortem format

The agent carries our template and standards as **skills** and **memory**: the
sections we use (timeline, root cause, impact, action items), our house style,
and lessons from earlier incidents worth comparing against. If we change how we
write postmortems, we edit that file and the very next draft follows suit.

### Give it the channel, the logs and GitHub

The agent reads through scoped **connectors**, connections Machine brokers on
the server so the model never handles a raw token:

- **The incident channel**: the stream of messages, who did what, and when the
incident began and ended.
- **The logs**: jumps in error rate and latency during the incident, ready to
place on the timeline.
- **GitHub**: deploys and merges from the same window, so it can match what
went out with when things went wrong.

It reads all three to rebuild the order of events. The only thing it writes is
the doc PR.

### Keep it a draft

What the agent produces is a **draft, not a decision**. Its doc PR is reviewed,
edited and merged by a person. It suggests a root cause and action items but
never settles them, and it doesn't publish a postmortem or hand out owners by
itself. Credentials are stored encrypted in the Secrets Manager and only handed
to the agents you grant them to.

### Open the draft for the team

From then on, every resolved incident produces a first draft with no one
asking. It includes a timeline pieced together from the channel and matched to
deploys and log spikes, a likely root cause with the evidence for it, and a set
of possible action items. The team opens the PR, fixes anything the agent got
wrong, adds what only they know, and merges a version they're happy to put
their names to.

> **The pattern**
> Resolving an incident on the **channel** trigger opens a session with
> read-only **connectors** to the channel, the logs and GitHub. Our format
> lives in **skills** and **memory**. The agent builds the timeline and opens a
> doc PR; people make it final.

## Guardrails

The agent reads the incident channel, logs and deploy history, then writes a
document people will act on. So its access is narrow and contained:

- **Isolation.** Each incident runs in its own sealed sandbox. The session
  reads the timeline, matches up deploys and logs, and sends nothing out except
  the draft doc PR.
- **Scoped secrets.** Keys for the channel, the logs and GitHub are stored
  encrypted in the Secrets Manager and only handed to the agents you grant them
  to.
- **PR-gated.** The postmortem arrives as a doc PR, not a published page. A
  person checks the timeline, reworks the root cause and action items, and
  decides when to merge.
- **Everything is code.** The postmortem format and each tool's permissions are
  files in the repo. They're versioned and only change through a reviewed
  **change request**, never a dashboard switch.

## The outcome

- **Every incident:** A first draft ready as soon as it's resolved
- **PR-gated:** The agent writes the draft; the team reviews it and makes it final
- **3 systems:** Incident channel, logs and GitHub, handled by a single agent

Now every resolved incident arrives with a postmortem draft waiting as a doc
PR, its timeline already matched to deploys and log spikes. The team puts its
energy into judging the cause and choosing the follow-ups instead of piecing
together what happened, and far fewer incidents close without a writeup.