# How we triage on-call alerts before anyone gets paged Our triage agent on Machine reads Sentry, our logs and GitHub. It puts together a first diagnosis for every alert, and only wakes a person when it can't settle the alert or the severity is high. Canonical page: https://themachine.myapps.ai/use-cases/oncall-triage It's 3am and your phone is buzzing. Before you can fix anything you have to get the trace up, see what shipped lately, dig through the logs and decide if this is a real fire or just smoke. Almost all of that is legwork, and you are doing it with no context and one eye open. It's hard to imagine a worse moment for it. So we let an agent on Machine take the first look. As soon as an alert goes off, it collects the stack trace, the latest deploys and the matching logs, posts an initial diagnosis in the incident channel, and only pages someone if it can't settle the alert or the severity is high. Here's how the pieces fit: what sets it off, how each run is isolated, and the limits we put on it. - **Team:** Machine - **Runs on:** Each alert, the moment it fires - **Connected systems:** Sentry · Logs · GitHub - **Mode:** Starts on each alert · pages a person only if needed ## The problem Nobody fixes anything in the opening minutes of an incident. Those minutes go on getting your bearings. What's the error? When did it begin? What went out right before? Is it hitting one customer or everyone? The engineer on call digs up each answer by hand, barely awake, before they can even tell whether the page deserved to wake them. The standard remedies only go so far. Page on every alert and the rotation drowns in noise until people tune the pager out. Raise the thresholds and the noise drops, but so do the real regressions. A runbook is useful, yet a human still has to be up to read it. None of these collects the evidence on your behalf. ## What we built Every alert wakes an agent on Machine. When Sentry fires, Machine gives the agent its own isolated cloud computer, or sandbox, that can look at Sentry, the logs and GitHub but change nothing. The agent grabs the stack trace, lists every deploy since the error first showed up, lines up the logs around the spike, and posts its first diagnosis to the incident channel. A person is paged only when the agent can't settle the alert or the severity is high. ## How it works ### Make the alert the starting gun Sentry sends a signed webhook to the project each time an alert goes off. Machine answers by starting a new **session**: a clean sandbox for that alert alone, preloaded with the alert's details. One alert gets one throwaway computer, so nothing leaks from one incident into the next, and alerts that arrive together are worked on side by side. ### Hand it the triage playbook Our way of triaging is kept as **skills** and **memory**, the playbook the agent opens on every run. It knows which services cry wolf, how a genuine regression differs from a familiar flaky test, the severity rules for paging, and earlier incidents along with what caused them. If an alert proves harmless, we note it, and next time the agent spots it straight away. ### Connect what triage needs The agent reaches each system through a **connector**, a narrow link whose keys stay on our servers, so the model never handles a raw token. With them, it can: - **Open the Sentry issue.** Stack trace, how often it's happening and when it was first seen, so it knows when the trouble started. - **Line up the logs.** It fetches the log lines around the spike and matches them against the trace. - **Look at recent deploys in GitHub.** The commits and PRs that went out just before the error began, to point at a probable cause. - **Post in the incident channel.** The diagnosis, the deploy it suspects and the evidence, all in one message. ### Set the limits The agent is **read-only**. It investigates Sentry, the logs and GitHub, but it never deploys, rolls back or changes a thing. High-severity alerts, and any it can't settle, page a person right away. The diagnosis travels with the page; it doesn't replace it. Credentials are stored encrypted in the **Secrets Manager** and only handed to the agents you grant them to. ### Let alerts do their own legwork Now each alert brings its own context. Within the first minute, the trace, the deploy window, the matching logs and a first diagnosis are sitting in the channel. A spike we know is harmless gets closed with the reasoning written out. A severe or unsettled alert pages the engineer on call with the digging already done, so they wake up to answers and not an empty terminal. > **The pattern** > A **trigger** on every alert starts a session with limited, read-only > **connectors** into Sentry, the logs and GitHub. Our triage playbook is kept > as **skills** and **memory**. The agent diagnoses first and wakes a person > only when it can't settle the alert or the severity is high. ## Guardrails This agent looks at production data, so what it can reach is tightly fenced: - **Isolation.** Each alert gets a sandbox to itself. Inside, the agent can pull traces, logs and deploy history to work out a diagnosis. The only things that leave are the channel message and any page. - **Scoped secrets.** The Sentry, logging and GitHub keys are stored encrypted in the **Secrets Manager** and only handed to the agents you grant them to, when the sandbox starts. - **Human approval gate.** Deploys and rollbacks are never the agent's call. Severe and unsettled alerts go to a person, and that person owns whatever happens next. - **Everything is code.** Configuration, skills and permissions are versioned files in the repo, changed only through a reviewed **change request**, never a dashboard toggle. ## The outcome - **First-pass:** A diagnosis posted before anyone's phone rings - **Less noise:** Harmless alerts closed with reasons instead of paging someone - **3 systems:** Sentry, logs and deploy history, handled by one agent The legwork at the start of an incident is done before anyone is woken. When the agent does page, the trace, the likely deploy and the matching logs come with it. The engineer on call gets to spend their time deciding and fixing, not fumbling for context in the dark.