You know the test. It goes red, nobody can say why, someone hits rerun, it turns green and the build carries on. Do that enough times and the team stops believing CI at all. Then one day a genuine bug fails that same test, and it gets waved through with the usual "ignore it, it's flaky". Fixing the worst offenders never makes it onto anyone's week, because no one can say which tests are really burning the most reruns.

So we gave the job to an agent on Machine. Once a day it reads through our CI history and gives each test a score for how often it changes result when the code did not change. The worst ones go into a quarantine PR, with a tracking issue alongside. It marks tests as skipped and nothing more: it will not delete a test and it will not merge its own PR. A person reviews the quarantine, and lifts it later once the test has been repaired.

The problem

Flaky tests are hard to see because the system covers for them. The job retries, the second attempt passes, the badge goes green, and the failure leaves no trace anyone looks at. Over a few months and a few hundred tests, a small group ends up costing you a rerun every single week. Worse, a test that is truly broken gets the same lazy retry as one that is merely unreliable.

The usual fixes miss. Retrying until green buries the evidence rather than counting it. A Slack thread where people ask "is that one acting up again?" only works if somebody spots it and remembers. Ripping out a flaky test loses whatever it was genuinely checking. And any manual approach starts with hours of scrolling old CI logs just to work out which tests are the real culprits.

What we built

Every day a schedule wakes up a single, long-lived agent session on Machine. The agent opens its running flakiness ledger, fetches whatever CI runs have landed in GitHub since it last looked, and refreshes each test's score. When a test crosses our quarantine line, the agent raises a PR that skips it, states why, and links to the failing runs. It also keeps one tracking issue up to date with every test currently in quarantine. Deleting tests and merging PRs are both off the table.

How it works

The pattern

One long-lived session gets a nudge from a daily cron, so the flakiness ledger is never lost between runs. The agent reads CI results from GitHub, scores and ranks each test, and produces just three things: a quarantine PR, a tracking issue and a Slack summary. Merging and deleting stay with people.

Guardrails

This agent affects which tests run in CI, so we keep its reach narrow and one-way:

• Isolation. Each run takes place inside the session's own sandbox. The only things that come out are the PR, the issue and the Slack message.
• Scoped secrets. Both the GitHub connector and the GH_TOKEN behind the gh tool are stored encrypted in the Secrets Manager and only handed to the agents you grant them to.
• Skip, never delete. The only change it makes to a test file is a quarantine marker that says why. Tests, their assertions and their files all stay where they are.
• PR-gated. It opens a PR plus an issue, then steps back. No merges, no pushes to the default branch. People own the merge and, later, the decision to release a test from quarantine.
• Everything is code. Scoring rules, skills and permissions live as files in the repo. They carry a version history and only change through a reviewed change request, never a toggle in some dashboard.

The outcome

The tests that quietly cost us a rerun each week now arrive in order, worst first, with a PR and a tracking issue for the biggest offenders. CI calms down, and we keep the coverage we never meant to throw away. When the team sits down to fix flakes, it starts with the ones doing the most damage.