Autonoma: agentic end-to-end testing

The number we saw everywhere was about seven concurrent agent worktrees before the shared environment became the bottleneck and bugs started reaching production. That is not a ceiling on how fast agents write code. It is a ceiling on how fast anyone can verify what they wrote, and what puts it there is that they are all testing against the same environment.

Agents write code faster than anyone can verify it. Green CI is no longer proof of safety: a coding agent produces a clean diff with a convincing description, tests pass, and the bug ships. Verifying that a change actually works requires a real environment (the app running, a real browser, real data flows) and nobody had solved the environment. Autonoma exists to lift that ceiling.

The way I put it internally: writing end-to-end tests by hand in 2026 is like writing assembly in 1990, and writing them with an AI is like writing assembly with an AI. It works. It is still madness. Meanwhile the industry's answer is outsourced QA, senior engineers losing a third of their week to a Cypress suite that broke on Tuesday for reasons nobody can name, green pipelines that are green because someone commented out the failures, and forty minutes of CI to find a bug a human would have seen in thirty seconds.

Product 1 (2024-25): a no-code tool letting QA people act as automation engineers. Record-and-replay against real browsers and devices. Took it from zero to ~$550K ARR with enterprise deals up to $50K ACV: a weekly regression pipeline at 98% accuracy for a major Argentine bank's digital brand, several fintechs, and scale-ups including Superhuman and Mercor. Fintech was the ICP, and at AIFI-25 in Buenos Aires (September 2025) Autonoma was named best AI startup for its innovation and impact on the financial ecosystem.

The decision (Feb 2026): delete it. Record-and-replay treats symptoms. It records what a user did, not what the app is supposed to do, so it can't tell you what to test when the code changes, and code was starting to change faster than anyone could re-record. QA as a job function is a shrinking market; the buyer is now the engineer, and the system has to be agent-first. I wrote about this in 18 Months of Code, Gone.

Product 2 (shipped Apr 2026): the platform reads the codebase, decides what to test on every PR, drives a real browser against an isolated environment, and reports only what matters. MCP-first; the UI is optional.

How it ended. We did not deliver the second product in time. That is the whole of it, and it was my call to make. We worked out that we could not finish on the cash we had, so we returned the money to our investors rather than spend the rest of it finding out.

The mistake was not spending. We were top-percentile on cash efficiency and I would not change how the money was used. The mistake was the plan it was used for: we raised $1.5M and set out to reach a Series A on it, when what we had raised was a seed. Nothing in the execution was going to close that gap, because the gap was in the arithmetic before we started. If I ran it again I would size the raise to the milestone, or size the milestone to the raise, and I would do that on day one rather than discover it eighteen months in.

So the honest frame for what follows: the first product was proven by customers paying for it, and the second one never got the chance to be. The architecture is real and it ran. It did not run for long enough.

Core loop. A GitHub app subscribes to PR and commit events. An impact-analysis agent reads the diff against every test spec and decides whether any test needs to run; nothing is built or deployed unless something changed that matters. Tests are markdown files with frontmatter (slug, immutable description, high-level steps), one folder per feature. The description is the deletion criterion: if a change invalidates the description itself, the test is removed, not patched.

Preview environments. Per-PR deployment of the customer's full stack into a network-isolated Kubernetes namespace with a full ephemeral database copy, thrown away with the PR. Tests never write into shared or production data, and run fully parallel with no collisions. The agent has access to logs, deployment state, environment variables and code, so it can check whether a failure is a real bug rather than guess from the screenshot.

This is the product difference, not a technical one. Previews that point at production data can't be tested: for a marketplace, the agent either never creates a listing, or real users see one. Split the data layer or don't test.

SDK. The first-generation SDK tried database introspection plus graph analysis to find the root table and synthesize test data. It failed for a structural reason: you can't infer business rules from a schema. The current SDK calls the customer's own creation functions and provisions a clean organization per test.

The browsing agent, and why it's cheap. Navigation paths are derived from the source code, not inferred by the model at runtime. Because the agent already knows where it's going, it can run on small models. Median cost is $0.50 per PR. A typical LLM code review costs $5-15 per PR and reports opinions about the diff; ours reports video, logs, screenshots, and the code path, from an actual execution. Vision models were evaluated for the browser task (UI-TARS, Qwen3-VL, Gemini baseline) with mutation-based ground truth: healthy vs. deliberately broken UI states, so the eval needs no LLM judge.

Reporter. Reads every test outcome across the run and folds duplicate findings into a single actionable report. The output scales with the number of distinct problems, not the number of tests.

Stack: TypeScript, Bun, Effect, Hono/tRPC, Postgres (multi-tenant RLS), Kubernetes, Temporal.

Flakiness is deliberately absent from the metrics, and that is the pitch. It is not tuned away, it is designed out: the environment isolation and the clean-organization-per-test SDK mean there is nothing for a test to collide with, and classification handles the residue. Every competitor fights noise and flakiness. We only ever had to fight noise. That is the whole reason the numbers below mean anything.

Against the noise, every outcome is classified: pass / scenario failure (missing data) / environment failure / outdated test / client bug. We found that a frontier model alone is not reliable at judging whether a failure is a real bug, hence the structured classification rather than a single verdict prompt.

Hand-labeled by the whole team across partner PRs, last measurement:

• ~95% of raw failures were not client bugs
• Of bugs we reported, 77% were real (precision), on customers whose setup is correct. That's the platform's ceiling today. The 98% figure for the bank above is a different measurement: accuracy of a fixed weekly regression suite on Product 1, where the tests were hand-authored and the environment was stable.

None of us had done marketing. We built the growth engine the same way we built the product: AI-generated content, experiments, keep what moves the number, kill what doesn't. Twelve months later: 0 to 5.9M search impressions and 68.8K clicks, peaking at 2.06M impressions and 24K clicks in a single month, driving ~1,400 signups a month.

The onboarding funnel after opening self-serve: ~1,400 signups/month → 350 → 10 → 3 paying. Healthy top, healthy bottom, 97% loss at one step. Users were screenshotting the onboarding into Claude to get through it, so the whole flow moved to MCP.

Nine engineers including the three technical founders, in an eleven-person company. I ran hiring end to end, roughly 100 interviews at Autonoma on top of roughly 100 at Google. The process is a simple coding challenge that reveals everything (I wrote about it: I Interviewed 200+ Engineers. Coding Challenges Are Still the Best Way to Hire.). The team includes engineers who have compiled models to hand-tuned x86 assembly for inference performance.

Not the code. The platform is source-available; the artifacts are copyable. The labeled corpus and the evaluation loop aren't. The commit history is deliberately bursty (implement, evaluate by hand, implement, evaluate) and the last two weeks before any release are an evaluation phase whose output is a published limitations page. Onboarding evaluation runs on the same loop.

That is the moat we had. The ones we were building toward were written down as bets, and a bet is a commitment: startups win by power law, and you do not reach the top of that curve by changing direction often enough to never be the best in the world at anything.

The browsing dataset. This is the one I still think about. Ninety-nine percent of the web sits behind a login, so a general model cannot navigate real SaaS: it has never seen any. We were on the inside of those products, because we read the customer's code to derive the navigation paths in the first place. Every run was therefore producing something nobody else could collect, a record of real application flows behind real authentication. Two or three years of that and it is the best SaaS navigation dataset in the world, at which point it is either an internal advantage or it is the product: the frontier model for navigation. The data was a byproduct of the thing we already had to do, which is the property you want in a moat.

The eval set. Build a bug-finding benchmark out of open source history. You know which PR introduced which bug because you can see the commit that fixed it, so you can replay the system against the moment before the fix and ask whether it would have caught it. That is the same harness the recall problem below needs, and it compounds: every point of accuracy it surfaces is a point your competitors cannot see about themselves.

Start with startups building in the future. They have the sharpest pain and the fewest alternatives, but that is not the reason. The reason is that they are deciding how software will be built, both the methodology (AI-native coding) and the shape of the product (apps built around AI workflows). Coupling to them is optimizing against a growing market rather than a shrinking one, and when everyone else works that way, you are already the vendor who is ready for them.

Open source distribution. Two halves: fund and maintain Autonoma's integration into open source repositories to stay close to the people who write the code, and ship boilerplates for each major stack with Autonoma already wired in, so that when a model is asked for a starter stack it recommends one that includes us. Distribution through the thing that now writes most of the code.

The real problem isn't false positives. It's "do I care?" The hard question moved from "is what we report false" to "is what we report worth a human's attention." Papercuts in volume drown the high-signal bugs, and there's no suppression mechanism yet.

Ambiguous findings. An invoicing customer's PR changed an invoice title on scheduling, and the PR description said so. Bug, or product iteration? Undecidable from the diff. A later PR swapped it for the username. Fix, or more iteration? Still undecidable. If the coding agent passed session intent ("let me schedule invoices"), we could challenge it: you changed a title nobody asked you to change. That's a new verdict class, ambiguous finding: ask, don't assert, and it needs the agent stack upstream to hand over intent.

Recall is unmeasured. Precision is 77%; I don't know what we miss. The harness is designed (backtest against historical fix commits in large, commit-rich repos to see whether we'd have caught the bug) and would take roughly a month funded. Closed-source customer repos differ materially from tidy open-source ones, which is the honest caveat on any number it produces.

Setup. A large share of customers have a partially broken configuration, which caps precision below the platform ceiling. That is why the recipe was moving off our backend and toward the customer's own environment.

Build-vs-buy on the sandbox. Firecracker microVMs were designed in detail (bare-metal EC2 for nested virtualization, Temporal orchestration, tiered volume cache across local disk / EBS / S3). At current demand every Firecracker flavor, managed or custom, costs more to build or run than the existing Kubernetes setup, so it ships on Kubernetes with network policies. I know container kernel-sharing is a sandbox-escape risk; it's an accepted one at this scale, not an unknown.