How we test every pull request without lifting a finger
Reading a diff catches the bugs you can see on the page. The rest only show up when the code actually runs. A test that's green on your laptop but flaky in CI. A route that's fine on the happy path and throws a 500 on an odd input. A migration that looks harmless but locks a table once real traffic hits it. Those usually turn up in staging, or worse in production, long after someone clicked approve.
Our QA agent on Machine does that running for us the moment a PR opens. Here is how we test our own changes with it, along with what it connects to and the limits we set.
The problem
Some bugs are invisible in a diff. A reviewer reads the code and signs off, yet nobody has pulled the branch, run the tests, called the new endpoint and seen what comes back. CI only runs the tests the author thought to write. It can't cover the cases they never pictured.
The standard answers each leave a hole. A green build proves the old tests still pass. It doesn't prove the new code is right. A manual QA pass is careful, but it's slow and shows up after review is done. An AI reviewer that only reads the diff can't run anything, so whatever breaks at runtime slips past it.
What we built
Every PR on Machine sets off an agent. When the PR opens, and again on each push, it gets an isolated session: a sandbox, which is a fresh cloud computer just for that run. The session has scoped access to the branch, the tests, a test environment we can deploy to, and the Cloudflare edge that sits in front of it. The agent checks out the change, runs the tests, tries the new behaviour on a live deploy, and posts a pass or fail on the PR.
How it works
Let GitHub start each run
GitHub sends a signed webhook to the project. Opening or updating a PR sets it off, and each time a new session starts in its own sandbox, already holding the branch being tested. One PR, one session, one throwaway machine. Nothing leaks between runs, and several PRs can be tested side by side.
Hand it the branch and the QA playbook
The session checks out the branch and does a clean install. How we like QA done is kept as skills and memory, the playbook the agent carries and improves: how to run the tests, which flows matter most, the edge cases that have bitten us before, and what a good report includes. When a bug gets through, we note it down and the agent uses it from the next run on.
Plug in what testing needs
Scoped connectors link the agent to each system. Machine keeps the tokens on its own servers, so the model never sees one. With them the agent can:
- Run every test. Unit, integration and end-to-end, all inside the sandbox, with the full error output saved.
- Deploy to the test environment. It puts the change on a short-lived deploy and drives it start to finish through the new paths.
- Check it at the edge with Cloudflare. It looks at how the change behaves through the edge (routing, headers, caching, redirects), not only on localhost.
- Report back in GitHub. The pass or fail, plus any way to reproduce a failure, lands on the PR as a check and a comment.
Draw the lines
The agent works only in the test environment. Production is off limits. It won't merge, and it won't deploy to prod. It hands back a result, and a person decides on the merge. Credentials are stored encrypted in the Secrets Manager, added at run time, and only handed to the agents you grant them to.
Get PRs that come already tested
Put together, a fresh PR now checks itself out, runs the tests, goes up on the test environment, gets tried through Cloudflare, and reports back. A pass shows green. A failure shows a red check with the command that broke, the logs and the steps to reproduce it. Flaky tests are called out with proof. A broken edge route gets caught before it ever reaches staging.
Summary
Every PR fires a trigger that opens a session with scoped connectors to the branch, the tests, the test environment and Cloudflare. Skills and memory hold the QA playbook. The agent never leaves the test environment, and a person makes the merge call.
Guardrails
The agent can reach the test environment and the edge, so that reach is kept tight and fenced in:
- Kept apart. Each PR gets its own sealed-off sandbox on its own branch. Inside, the session may install, deploy and poke at the change to recreate a failure. Only the report it posts goes back out.
- Locked-down keys. The GitHub, test environment and Cloudflare credentials are stored encrypted in the Secrets Manager, added to the sandbox at run time, and only handed to the agents you grant them to.
- Test environment, full stop. Its access ends at the test layer. No production, no prod deploys, no merging. It reports, and your team decides.
- All of it lives in code. The agent's setup, skills and permissions are files in the repo. They're versioned, and you change them with a reviewed change request, not a setting on a dashboard.
The outcome
The runtime bugs a diff hides now appear on the PR as soon as it opens, along with the command that failed, the logs and how to reproduce it. Reviewers get to think about design instead of pulling branches by hand. And whatever reaches staging has already run on the test environment first.
More use cases
How we answer GDPR data-subject requests before the deadline
Our Machine agent checks each GDPR access or deletion request, finds the person's data across our product database, builds the report well within the SLA and hands it to legal before anything is sent.
How we keep our docs matching what the code does
Once a day a Machine agent reads every commit merged since its last pass, finds the doc pages those changes made wrong, rewrites them and sends you one pull request to review.
How we run day-to-day operations from a Slack thread
One agent you can call from Slack, with narrow access to our database, Stripe, Linear and GitHub. Describe the job in a thread and it carries it out across every tool involved.
Put your first agentto work today.
- SSO
- RBAC
- Governed agents
- Full audit trail