How we monitor our data pipelines and warehouse health
Data pipelines have a nasty habit of breaking without a sound. A load job stops adding rows and says nothing. Someone upstream renames a column and half your fields arrive empty. A table meant to update hourly is still showing yesterday. No error pops up anywhere. The dashboards simply start telling fibs, and the first person to notice is usually whoever is presenting the wrong number to a room full of stakeholders.
So we put an agent on Machine in charge of checking the warehouse directly, once an hour. Is each table as up to date as its SLA promises? Does today's row count fit the usual pattern? Has a schema shifted under someone's feet? Did a load job fall over? The agent only ever reads. When something looks off, it posts in the data team's Slack channel and drafts a GitHub issue with its best guess at the cause. This is exactly how we keep watch over our own warehouse.
The problem
A sick warehouse rarely tells you so. A load job designed to skip a bad batch and keep going can fail without raising any error at all. A table can go stale because someone switched off a cron upstream, with nothing crashing anywhere. A source API can add a field, and the load still reports success while quietly dropping or blanking the bits it does not understand. From the outside, all of these look perfectly healthy right up until a query further down returns nonsense.
The usual safety nets have holes. A dashboard sitting on the data cannot spot that the data itself is broken: it draws whatever it is given, right or wrong. A "did the job run?" check proves the process finished cleanly, not that the table it filled makes any sense. A person scanning row counts in a spreadsheet will get there in the end, typically after a bad report has already gone out. This is also not the same as an app crashing mid-request. The pipeline can succeed on paper while the warehouse is still unwell.
What we built
Every hour, a cron on Machine starts an agent. It opens a fresh session that can read the warehouse but cannot change it. For each table we watch, the agent checks freshness against the table's SLA, compares today's row count with that table's own recent history, holds the current schema up against the shape it saw last time, and hunts for loads that failed or quietly stopped. If anything falls outside the limits, it sends the data team's Slack channel an alert naming the table, the problem and the most likely cause, and it drafts a GitHub issue with the same diagnosis so the incident has a home. It never writes to the warehouse. The alert and the draft issue are all it produces.
How it works
Schedule an hourly run
A cron trigger starts the agent once every hour. Each start creates a brand-new session: a fresh cloud computer (sandbox) that exists for that one run and is then thrown away. Every hour gets its own run, so every check is worked out from the warehouse as it is right now, and nothing is carried over from the hour before.
Teach the agent what healthy looks like
The agent's playbook sits in skills and memory and goes wherever the agent goes. It defines what counts as stale, what a normal row-count pattern looks like and what shape each schema should have. It holds a freshness SLA for every table, notes which tables grow in spurts and which grow evenly, and keeps a record of past incidents to compare new problems against. If we tighten an SLA, or discover that a table legitimately goes quiet once a week, we write that down and the checks follow.
Give it read-only access to the warehouse
A scoped connector, meaning a narrow connection whose credential stays on our servers and never reaches the model, lets the agent read the warehouse directly. It looks at:
- Freshness: the latest load time for each table we watch, compared with that table's SLA.
- Row counts: today's total next to the table's own recent history, to spot a load that came up short or ran twice.
- Schemas: the current columns and their types, compared with the shape seen last time, so drift is caught before a query further down breaks.
- Load history: recent job runs, to find any load that failed or quietly stalled.
Raise the alarm, never the wrench
If a check lands outside the limits, the agent posts in Slack with the table, what looks wrong and its best guess at why. It also drafts a GitHub issue with the same diagnosis, using a narrowly scoped token, so the incident becomes a draft someone can track, never a merged change. The pipeline, the schema and the data are all left exactly as they were.
Draw the lines
Across the whole warehouse, the agent can only read. No table, schema or pipeline setting is open to it for writing. It watches and it reports. It produces exactly two things: the Slack alert and the draft GitHub issue. Credentials are stored encrypted in the Secrets Manager and only handed to the agents you grant them to, at the moment a run starts.
Let the warehouse keep an eye on itself
Once this is live, every hour brings a fresh sweep of the warehouse. Freshness is held against the SLA, row counts against their pattern, schemas against their last known shape, and loads are checked for failures. Anything outside the limits lands in Slack within the hour, with a GitHub issue already drafted and the probable cause spelled out. The data team decides the next move. Nothing changes by itself.
The pattern
An hourly cron creates a session with a read-only connector into the Postgres warehouse. SLAs and normal patterns are stored as skills and memory. The agent can see everything, and the only things it writes are a Slack alert and a draft GitHub issue.
Guardrails
This agent can look at every corner of the warehouse, so its access is narrow and runs in one direction only:
- Isolation. Each run happens in a sandbox of its own. The session can reach only the warehouse it was scoped to, and the Slack alert and draft issue are the only things that come back out.
- Scoped secrets. The warehouse credential and the GitHub token are stored encrypted in the Secrets Manager and only handed to the agents you grant them to, injected into the sandbox when the run starts.
- Read-only, no exceptions. The warehouse connector cannot write. The agent has no way to edit a row, change a schema or touch a pipeline. Alerting and drafting are its whole job.
- Draft, not decide. The GitHub issue arrives as a draft with a diagnosis. It is never closed or assigned automatically. A person takes ownership of the incident from that point.
- Everything is code. The SLAs, the normal patterns and each system's permissions are files in the repo. They keep a version history and change through a reviewed change request, not a setting in a dashboard.
The outcome
Warehouse trouble used to reveal itself as a strange number on someone's dashboard. Now it turns up in an hourly health check that names the table, the problem and the likely cause, already posted in Slack and already drafted as an issue. The agent reads and reports. The data team chooses what to fix.
More use cases
How we answer GDPR data-subject requests before the deadline
Our Machine agent checks each GDPR access or deletion request, finds the person's data across our product database, builds the report well within the SLA and hands it to legal before anything is sent.
How we keep our docs matching what the code does
Once a day a Machine agent reads every commit merged since its last pass, finds the doc pages those changes made wrong, rewrites them and sends you one pull request to review.
How we test every pull request without lifting a finger
Our Machine QA agent works with GitHub and our test environment. For each pull request it checks out the branch, runs the tests, tries the change for real, and posts what it found.
Put your first agentto work today.
- SSO
- RBAC
- Governed agents
- Full audit trail