# How we test every pull request without lifting a finger

Our Machine QA agent works with GitHub and our test environment. For each pull request it checks out the branch, runs the tests, tries the change for real, and posts what it found.

Canonical page: https://themachine.myapps.ai/use-cases/qa-agent

Reading a diff catches the bugs you can see on the page. The rest only show
up when the code actually runs. A test that's green on your laptop but flaky
in CI. A route that's fine on the happy path and throws a 500 on an odd
input. A migration that looks harmless but locks a table once real traffic
hits it. Those usually turn up in staging, or worse in production, long after
someone clicked approve.

Our QA agent on Machine does that running for us the moment a PR opens. Here
is how we test our own changes with it, along with what it connects to and
the limits we set.

- **Team:** Machine
- **Runs on:** Each PR, when it opens and on every push
- **Connected systems:** GitHub · Test environment · Cloudflare
- **Mode:** Started by events · results posted on the PR

## The problem

Some bugs are invisible in a diff. A reviewer reads the code and signs off,
yet nobody has pulled the branch, run the tests, called the new endpoint and
seen what comes back. CI only runs the tests the author thought to write. It
can't cover the cases they never pictured.

The standard answers each leave a hole. A green build proves the old tests
still pass. It doesn't prove the new code is right. A manual QA pass is
careful, but it's slow and shows up after review is done. An AI reviewer that
only reads the diff can't run anything, so whatever breaks at runtime slips
past it.

## What we built

Every PR on Machine sets off an agent. When the PR opens, and again on each
push, it gets an isolated session: a sandbox, which is a fresh cloud computer
just for that run. The session has scoped access to the branch, the tests, a
test environment we can deploy to, and the Cloudflare edge that sits in front
of it. The agent checks out the change, runs the tests, tries the new
behaviour on a live deploy, and posts a pass or fail on the PR.

## How it works

### Let GitHub start each run

GitHub sends a signed webhook to the project. Opening or updating a PR sets
it off, and each time a new **session** starts in its own sandbox, already
holding the branch being tested. One PR, one session, one throwaway machine.
Nothing leaks between runs, and several PRs can be tested side by side.

### Hand it the branch and the QA playbook

The session checks out the branch and does a clean install. How we like QA
done is kept as **skills** and **memory**, the playbook the agent carries and
improves: how to run the tests, which flows matter most, the edge cases that
have bitten us before, and what a good report includes. When a bug gets
through, we note it down and the agent uses it from the next run on.

### Plug in what testing needs

Scoped **connectors** link the agent to each system. Machine keeps the tokens
on its own servers, so the model never sees one. With them the agent can:

- **Run every test.** Unit, integration and end-to-end, all inside the
sandbox, with the full error output saved.
- **Deploy to the test environment.** It puts the change on a short-lived
deploy and drives it start to finish through the new paths.
- **Check it at the edge with Cloudflare.** It looks at how the change
behaves through the edge (routing, headers, caching, redirects), not only
on localhost.
- **Report back in GitHub.** The pass or fail, plus any way to reproduce a
failure, lands on the PR as a check and a comment.

### Draw the lines

The agent works **only in the test environment**. Production is off limits.
It won't merge, and it won't deploy to prod. It hands back a result, and a
person decides on the merge. Credentials are stored encrypted in the Secrets
Manager, added at run time, and only handed to the agents you grant them to.

### Get PRs that come already tested

Put together, a fresh PR now checks itself out, runs the tests, goes up on
the test environment, gets tried through Cloudflare, and reports back. A pass
shows green. A failure shows a red check with the command that broke, the
logs and the steps to reproduce it. Flaky tests are called out with proof.
A broken edge route gets caught before it ever reaches staging.

> **Summary**
> Every PR fires a **trigger** that opens a session with scoped
> **connectors** to the branch, the tests, the test environment and
> Cloudflare. **Skills** and **memory** hold the QA playbook. The agent never
> leaves the test environment, and a person makes the merge call.

## Guardrails

The agent can reach the test environment and the edge, so that reach is kept
tight and fenced in:

- **Kept apart.** Each PR gets its own sealed-off sandbox on its own branch.
  Inside, the session may install, deploy and poke at the change to recreate
  a failure. Only the report it posts goes back out.
- **Locked-down keys.** The GitHub, test environment and Cloudflare
  credentials are stored encrypted in the Secrets Manager, added to the
  sandbox at run time, and only handed to the agents you grant them to.
- **Test environment, full stop.** Its access ends at the test layer. No
  production, no prod deploys, no merging. It reports, and your team decides.
- **All of it lives in code.** The agent's setup, skills and permissions are
  files in the repo. They're versioned, and you change them with a reviewed
  **change request**, not a setting on a dashboard.

## The outcome

- **Every PR:** Tested, deployed and tried out before a person reviews it
- **Pre-merge:** Runtime bugs caught well before staging sees them
- **4 systems:** The branch, the tests, the test environment and the edge, one agent

The runtime bugs a diff hides now appear on the PR as soon as it opens, along
with the command that failed, the logs and how to reproduce it. Reviewers
get to think about design instead of pulling branches by hand. And whatever
reaches staging has already run on the test environment first.