# How we catch flaky tests and put them in quarantine Our Machine agent reads GitHub CI history daily, ranks each test by how often it flips on unchanged code, then opens a quarantine PR and a tracking issue for a person to review. Canonical page: https://themachine.myapps.ai/use-cases/flaky-test-triage You know the test. It goes red, nobody can say why, someone hits rerun, it turns green and the build carries on. Do that enough times and the team stops believing CI at all. Then one day a genuine bug fails that same test, and it gets waved through with the usual "ignore it, it's flaky". Fixing the worst offenders never makes it onto anyone's week, because no one can say which tests are really burning the most reruns. So we gave the job to an agent on Machine. Once a day it reads through our CI history and gives each test a score for how often it changes result when the code did not change. The worst ones go into a quarantine PR, with a tracking issue alongside. It marks tests as skipped and nothing more: it will not delete a test and it will not merge its own PR. A person reviews the quarantine, and lifts it later once the test has been repaired. - **Team:** Machine - **Runs on:** Once a day on a schedule, in the same long-lived session - **Connected systems:** GitHub · CI run history · Slack - **Mode:** Opens a quarantine PR and a tracking issue; no merges, no deletions ## The problem Flaky tests are hard to see because the system covers for them. The job retries, the second attempt passes, the badge goes green, and the failure leaves no trace anyone looks at. Over a few months and a few hundred tests, a small group ends up costing you a rerun every single week. Worse, a test that is truly broken gets the same lazy retry as one that is merely unreliable. The usual fixes miss. Retrying until green buries the evidence rather than counting it. A Slack thread where people ask "is that one acting up again?" only works if somebody spots it and remembers. Ripping out a flaky test loses whatever it was genuinely checking. And any manual approach starts with hours of scrolling old CI logs just to work out which tests are the real culprits. ## What we built Every day a schedule wakes up a single, long-lived agent session on Machine. The agent opens its running flakiness ledger, fetches whatever CI runs have landed in GitHub since it last looked, and refreshes each test's score. When a test crosses our quarantine line, the agent raises a PR that skips it, states why, and links to the failing runs. It also keeps one tracking issue up to date with every test currently in quarantine. Deleting tests and merging PRs are both off the table. ## How it works ### Wake the same session every day A **cron trigger** goes off daily and points at one existing **session** (a cloud computer that stays put), rather than spinning up a new one each time. Since the session is reused, each test's track record carries forward. Scores build on weeks of history instead of starting over, so they reflect a pattern, not one bad morning. ### Hand the agent its playbook The agent carries our method with it as **skills** and **memory**: the way we calculate a flakiness score, the skip syntax for each test framework, and what belongs in a quarantine PR or the tracking issue. If we move the threshold, or discover why a particular test misbehaves, we note it down, and the next run already knows. ### Plug in the tools it reads and writes The agent works through a scoped **connector**, a connection to GitHub that Machine brokers on the server so the model never sees a raw token. A **GH_TOKEN** secret also lets it use the `gh` command line tool. With those it can: - **Pull CI results from GitHub**: per-test outcomes for each workflow run across a rolling window, to find tests that change result when the code stayed the same. - **Raise a quarantine PR on GitHub**: skip markers on the top offenders, each one explained and linked to the runs that prove it. - **Keep a tracking issue on GitHub**: a single living issue showing each quarantined test, its score and where it stands. - **Report in Slack**: what moved this run, meaning fresh quarantines, tests that are still unreliable, and tests that look ready to come back. ### Draw the lines it cannot cross Skipping and labelling are the agent's only moves; deletion is not one of them. It opens its PR and its issue, then hands over. It does not merge, and it cannot push to the default branch. Its keys are stored encrypted in the **Secrets Manager** and only handed to the agents you grant them to. ### Let it run the daily check From then on, the routine looks after itself. Each day the agent re-scores every test against fresh CI results. A test that crosses the line lands in a quarantine PR with its evidence, joins the tracking issue, and shows up in that day's Slack summary. Someone on the team reads the PR and merges it if the quarantine makes sense, then removes the skip when the test has been properly fixed. > **The pattern** > One long-lived **session** gets a nudge from a daily **cron**, so the > flakiness ledger is never lost between runs. The agent reads CI results from > GitHub, scores and ranks each test, and produces just three things: a > quarantine PR, a tracking issue and a Slack summary. Merging and deleting > stay with people. ## Guardrails This agent affects which tests run in CI, so we keep its reach narrow and one-way: - **Isolation.** Each run takes place inside the session's own sandbox. The only things that come out are the PR, the issue and the Slack message. - **Scoped secrets.** Both the GitHub connector and the `GH_TOKEN` behind the `gh` tool are stored encrypted in the Secrets Manager and only handed to the agents you grant them to. - **Skip, never delete.** The only change it makes to a test file is a quarantine marker that says why. Tests, their assertions and their files all stay where they are. - **PR-gated.** It opens a PR plus an issue, then steps back. No merges, no pushes to the default branch. People own the merge and, later, the decision to release a test from quarantine. - **Everything is code.** Scoring rules, skills and permissions live as files in the repo. They carry a version history and only change through a reviewed **change request**, never a toggle in some dashboard. ## The outcome - **Every day:** Fresh flakiness scores for each test, based on the newest CI runs - **Ranked, not guessed:** Tests picked for quarantine on evidence rather than hunches - **Human merge:** The agent suggests a quarantine; your team makes the call The tests that quietly cost us a rerun each week now arrive in order, worst first, with a PR and a tracking issue for the biggest offenders. CI calms down, and we keep the coverage we never meant to throw away. When the team sits down to fix flakes, it starts with the ones doing the most damage.