Building software with a team of AI agents: what stays with the human
In recent months we had software built by teams of AI agents on several projects: an architect, a builder per app, a security reviewer and a tester. It works, and faster than we expected. Not because the agents are so clever. It works because of the structure around them, and because it is settled which decisions stay with a human.
Roles with their own assignment
One agent that does everything checks its own work with the same blind spot. So we split. The architect fixes the contract: the shared layer, the data model and what each app may import. Builders each work on their own part of the repo. Security and QA are explicitly told to find mistakes in someone else's work. The contract is on the main branch before parallel sessions start, so nobody builds on an assumption.
Every agent has its own context, and that costs tokens. A team uses considerably more than a single session. We keep five team members per session as the ceiling.
Review finds real things
A review agent is not a formality. In recent weeks they found, among other things, an exception on a secure cookie that depended on a hostname alone, test runs that could fall back to real access codes from the environment, and write routes that could be called from another site. For an infrastructure change we ran four correction rounds, including a premortem: suppose this fails a month from now, what caused it? Only then did the change go ahead.
Sometimes the first finding is in the own work of the agent that built the check. That is exactly the point.
The memory lives in the repo
An agent starts every session blank. What was decided yesterday only exists today if it was written down. So we keep a decision log per project, one line per decision: date, what, why. Lines are never rewritten.
- A temporary decision carries its expiry condition in the line itself, written so a reader can check it unaided. Otherwise, a month later it looks like a permanent one.
- When a new line contradicts an older one, the older one gets a pointer to the new one. Otherwise the oldest version of every fact stays unchallenged the longest.
- Versions are pinned after checking the registry, not from the model's memory. One example: the newest TypeScript turned out not to work with the framework's type check, so we pinned the previous version, noting when to look again.
Tests on what a person sees
Agents easily write tests that confirm their own implementation. We ask for tests that measure what a user sees: end to end on a small iPhone (375 pixels wide), an Android phone and a laptop. Leak tests use canary words in restricted text, and a test has to be able to fail, otherwise it proves nothing.
And when a test is flaky, suspect the measurement first. More than once the instability was in how we measured, not in the app.
What stays with the human
- Policy and exceptions: what is confidential, who sees which names. The agent sets out the options, the owner chooses, and the choice goes into the log with its date.
- Production: releasing to production is blocked in the agent settings. Every change names its risks and its way back before the push.
- What we promise the client, such as where data lives. We write down what we can deliver, caveats included, so nobody promises more.
- Texts for people in distress. When a chatbot refers someone to a helpline, a person at the client reviews that text.
One caveat: blocks in agent settings are rules on the start of a command. A different spelling gets past them. The real protection is that the development environment holds no real secrets.
What it delivers
Speed, but above all traceability. Every change has a why, every decision a date and every risk a name. What it costs: tokens, and the owner's attention at the moments something needs deciding. You don't save on review; you move it to where it pays off.