How we triage on-call alerts before anyone gets paged
It's 3am and your phone is buzzing. Before you can fix anything you have to get the trace up, see what shipped lately, dig through the logs and decide if this is a real fire or just smoke. Almost all of that is legwork, and you are doing it with no context and one eye open. It's hard to imagine a worse moment for it.
So we let an agent on Machine take the first look. As soon as an alert goes off, it collects the stack trace, the latest deploys and the matching logs, posts an initial diagnosis in the incident channel, and only pages someone if it can't settle the alert or the severity is high. Here's how the pieces fit: what sets it off, how each run is isolated, and the limits we put on it.
The problem
Nobody fixes anything in the opening minutes of an incident. Those minutes go on getting your bearings. What's the error? When did it begin? What went out right before? Is it hitting one customer or everyone? The engineer on call digs up each answer by hand, barely awake, before they can even tell whether the page deserved to wake them.
The standard remedies only go so far. Page on every alert and the rotation drowns in noise until people tune the pager out. Raise the thresholds and the noise drops, but so do the real regressions. A runbook is useful, yet a human still has to be up to read it. None of these collects the evidence on your behalf.
What we built
Every alert wakes an agent on Machine. When Sentry fires, Machine gives the agent its own isolated cloud computer, or sandbox, that can look at Sentry, the logs and GitHub but change nothing. The agent grabs the stack trace, lists every deploy since the error first showed up, lines up the logs around the spike, and posts its first diagnosis to the incident channel. A person is paged only when the agent can't settle the alert or the severity is high.
How it works
Make the alert the starting gun
Sentry sends a signed webhook to the project each time an alert goes off. Machine answers by starting a new session: a clean sandbox for that alert alone, preloaded with the alert's details. One alert gets one throwaway computer, so nothing leaks from one incident into the next, and alerts that arrive together are worked on side by side.
Hand it the triage playbook
Our way of triaging is kept as skills and memory, the playbook the agent opens on every run. It knows which services cry wolf, how a genuine regression differs from a familiar flaky test, the severity rules for paging, and earlier incidents along with what caused them. If an alert proves harmless, we note it, and next time the agent spots it straight away.
Connect what triage needs
The agent reaches each system through a connector, a narrow link whose keys stay on our servers, so the model never handles a raw token. With them, it can:
- Open the Sentry issue. Stack trace, how often it's happening and when it was first seen, so it knows when the trouble started.
- Line up the logs. It fetches the log lines around the spike and matches them against the trace.
- Look at recent deploys in GitHub. The commits and PRs that went out just before the error began, to point at a probable cause.
- Post in the incident channel. The diagnosis, the deploy it suspects and the evidence, all in one message.
Set the limits
The agent is read-only. It investigates Sentry, the logs and GitHub, but it never deploys, rolls back or changes a thing. High-severity alerts, and any it can't settle, page a person right away. The diagnosis travels with the page; it doesn't replace it. Credentials are stored encrypted in the Secrets Manager and only handed to the agents you grant them to.
Let alerts do their own legwork
Now each alert brings its own context. Within the first minute, the trace, the deploy window, the matching logs and a first diagnosis are sitting in the channel. A spike we know is harmless gets closed with the reasoning written out. A severe or unsettled alert pages the engineer on call with the digging already done, so they wake up to answers and not an empty terminal.
The pattern
A trigger on every alert starts a session with limited, read-only connectors into Sentry, the logs and GitHub. Our triage playbook is kept as skills and memory. The agent diagnoses first and wakes a person only when it can't settle the alert or the severity is high.
Guardrails
This agent looks at production data, so what it can reach is tightly fenced:
- Isolation. Each alert gets a sandbox to itself. Inside, the agent can pull traces, logs and deploy history to work out a diagnosis. The only things that leave are the channel message and any page.
- Scoped secrets. The Sentry, logging and GitHub keys are stored encrypted in the Secrets Manager and only handed to the agents you grant them to, when the sandbox starts.
- Human approval gate. Deploys and rollbacks are never the agent's call. Severe and unsettled alerts go to a person, and that person owns whatever happens next.
- Everything is code. Configuration, skills and permissions are versioned files in the repo, changed only through a reviewed change request, never a dashboard toggle.
The outcome
The legwork at the start of an incident is done before anyone is woken. When the agent does page, the trace, the likely deploy and the matching logs come with it. The engineer on call gets to spend their time deciding and fixing, not fumbling for context in the dark.
More use cases
How we answer GDPR data-subject requests before the deadline
Our Machine agent checks each GDPR access or deletion request, finds the person's data across our product database, builds the report well within the SLA and hands it to legal before anything is sent.
How we keep our docs matching what the code does
Once a day a Machine agent reads every commit merged since its last pass, finds the doc pages those changes made wrong, rewrites them and sends you one pull request to review.
How we test every pull request without lifting a finger
Our Machine QA agent works with GitHub and our test environment. For each pull request it checks out the branch, runs the tests, tries the change for real, and posts what it found.
Put your first agentto work today.
- SSO
- RBAC
- Governed agents
- Full audit trail