AI agent supervision interview: Hellomatik’s Isaac Correa

AI agent supervision interview: Hellomatik’s Isaac Correa

AI agent supervision interview: why instructions don’t steer agents — guards do

He wrote the rule. The agents ignored it. In this AI agent supervision interview, Hellomatik co-founder Isaac Correa explains what eight months of his own coding-agent logs showed: written instructions barely moved behaviour, small programs that intercept an agent’s next action did — and a vendor’s model swap changed how often the rules broke overnight.

Hellomatik – A Madrid-based Company

Hellomatik is a Madrid-based company that builds software for complex tasks inside organisations. Its co-founder, Isaac Correa, is not a trained programmer, which turns out to matter to the argument: he runs many agent sessions at once and cannot watch any of them closely enough to catch a mistake in time.

That is the condition most of this AI agent supervision interview describes. If you are supervising agents at a distance, and by volume rather than by attention, then instructions you write once and hope will hold are a weak control — and the question becomes what actually enforces your intent. His answer, guards, is a concrete engineering pattern rather than a management philosophy.

Isaac Correa – co-founder of Hellomatik

Isaac Correa is a co-founder of Hellomatik, a Madrid company building software for complex tasks inside companies. He did not train as a programmer, and he typically runs several coding-agent sessions at once — which is precisely the situation in which supervision by reading does not scale.

His working paper publishes eight months of his own agents’ logs and examines a mistake he says he made himself: trying to steer agents with instructions when the instructions barely changed what they did. What kept them in line was a set of guards — small programs that inspect what an agent is about to do, check it against his rules, and stop it before it runs, telling the agent why. He reports that one writing rule, as a written instruction alone, had no effect he could detect; the share of drafts breaking it fell to roughly a sixth of its earlier level only after he strengthened the wording and added a guard, and he flags that he changed both at once and so cannot say how much each contributed.

He also reports that the model mattered as much as his rules: in the week a new model was released, 18% of its first drafts broke his writing rules against 57% for the previous model, and the gap held within the same sessions. Supervision, he found, never shrank — it stayed between a third and a half of his messages throughout, but shifted from fixing the agents’ work to nudging them to continue. The study covers one operator and one set of tools, and the paper and his two related essays are self-published and not peer reviewed.


Instructions versus guards

Q1. In one line, why do written instructions barely change what an agent does?

IC: Because a rule that’s only written down is one more thing the agent reads among many, and in my experience a mistake that’s only forbidden in writing comes back.

Q2. What is a guard, technically — where does it sit in the loop, what does it inspect, and what does it cost in latency or friction?

IC: A guard is a small program that checks the agent at the two moments I’d want a colleague to stop and think: just before it does something, and just before it tells me it’s finished. Claude Code, the tool I used, lets you plug a program into exactly those points. The guard reads what the agent is about to do, a command or the text of its answer, and if it breaks one of my rules, it blocks it and tells the agent why in one line, so it can try again. Most of my blocks came from three guards: one for destructive or forbidden commands, one for my writing rules, and one for claims that a task was finished when nothing showed it had been checked.

What does it cost? Almost nothing in time: I timed my writing guard on a long real conversation and it took under a tenth of a second. The real cost is the retry: my writing guard blocked about one turn in eleven, and each block meant one more pass. Worse than that is the wrong block. A guard that stops good work gets switched off, and then it protects nothing. That’s why, if one of my guards breaks, it lets the agent carry on instead of blocking everything.

The surprise was somewhere else. The guard reads the conversation from disk, and the tool’s public issue tracker documents that the agent’s last message may not be written yet when the guard looks. Of the mistakes my writing guard read and still let through, nine in ten were hiding in that last message.

The evidence

Q3. As a written instruction alone, one writing rule had no effect you could detect. How did you measure that, and what would a false negative have looked like?

IC: I didn’t trust the guard’s own log for that, because a log only starts the day the guard does. So I took the guard’s rules and ran them, unchanged, over every first draft I still had, months of them, including the weeks before the guard existed. A first draft is what the agent writes before anything has checked it, which is where its real habits show.

Then I tried to break my own instrument. Run side by side with the guard’s own code on 3,000 random answers, it flagged exactly the same 618; with one rule removed on purpose, it disagreed on 475. I do that because more than once what looked like a finding about my agents had turned out to be a flaw in my measurement. If a measure doesn’t move when you break it, it isn’t measuring anything.

The rule itself was simple: answer me in Spanish. With no rule, about one draft in six came back in English (16.2%); once the rule was written down, about one in ten (9.5%), and that period even includes two days after I had restated it more firmly. That looks like progress, but the period before the rule lasted only three days, and the statistics couldn’t tell the drop apart from chance: the odds ratio was 0.54, with a 95% interval from 0.28 to 1.06, and an interval that reaches 1 includes no effect at all. I won’t pretend otherwise. That’s exactly what a false negative would look like: a real but modest effect, too small to see in a short window. So all I can say is that I couldn’t detect an effect, which is different from saying there was none. What I could see clearly was the next step. Once I wrote the rule more firmly and put a guard behind it, fewer than one draft in thirty came back in English (2.7%).

Wording and a Guard

Q4. You strengthened the wording and added a guard at the same time, and say so yourself. How would you redesign the study to isolate each one?

IC: I firmed up the wording and switched on the guard two days apart, so I’m the first to say I can’t separate them. Next time I’d change one thing at a time, on one pinned model version, and try every combination, plain wording with and without a guard and firmer wording with and without one, with chance deciding which session gets which instead of me. In the sessions without a guard I’d still run it in shadow mode, writing down what it would have blocked without blocking anything, so every group is measured the same way. And I’d give the starting period several weeks. The analysis scripts are published with the paper, so anyone who wants to try that design on their own logs has somewhere to start.

Model-swap risk

Q5. Eighteen per cent of first drafts broke your rules on the new model against 57% on the previous one, and the gap held within the same sessions. What does that imply for someone buying AI software?

IC: That when you buy AI software, you’re also buying a model you don’t control. In the week a new one came out, the same rules, on the same days, were broken about three times less often, and the writing rules being compared hadn’t changed. Only three sessions used both models, which is a small number, but in those three the gap was just as wide: 55% against 17%. This time the change went my way. The next one might not, and no vendor writes its release notes in terms of your rules.

And there’s a less obvious risk. If I had edited my rules that week, I’d have been sure my edit did it. How many teams are congratulating themselves right now for an improvement the vendor shipped? So I’d ask any vendor which model sits underneath, whether they can hold it fixed, how much notice they give before changing it, and whether I can run my own checks before the change reaches my people. The model is the one part of the product that can change without asking you.

Test for Risk Before Signing

Q6. If a buyer wanted to test for that risk before signing, what would you have them do?

IC: I’d have them test it on their own cases, not the vendor’s, because a demo only shows what the vendor chose to show. Take a few dozen real cases from your own work and write down the rules your company actually cares about in a form a program can check: the language of a reply, a field that must never be filled in, a figure that must match its source, an action that must never happen without approval. Run the product on them and ask to see the first drafts as well as the polished answers, because an answer can look clean only because something cleaned it on the way.

Then make the test fail on purpose. Slip in a case you know is wrong and make sure the check catches it. And put in the contract that you can run it again every time the vendor changes the model. It’s what I do with my own agents: I don’t trust a test I haven’t seen fail.

Supervision Stayed

Q7. Supervision stayed between a third and a half of your messages, but shifted from fixing the agents’ work to nudging them to keep going. What does that shift actually feel like, and what does it imply about the work?

IC: It feels like going from proofreading to being a foreman. In the first months I read everything my agents wrote and fixed it. By the end, corrections were under a tenth of what I wrote, and one message in five was some version of “carry on”. The agents would stop to ask me something they could decide alone, or finish one batch of a long task and wait. Both habits ended up with a guard of their own, because a habit I only complain about comes back.

What didn’t change was how much of me it took. Supervision stayed between a third and a half of everything I wrote, and from about seven sessions upwards my pace levelled off at around twenty messages an hour, so each new session got a thinner slice of me. The number of busy sessions partly follows my own activity, so I wouldn’t lean too hard on that figure, but it matches how it felt. My agents freed my hands, not my attention.

In “The Last Tool” I wrote that people will stop operating their tools and start directing them. This is what that looks like from the inside. The skill that’s left is knowing a bad answer when you see one, and I’ll keep directing for as long as I have it.

Limits and generalisation

Q8. One operator, one set of tools. What do you think generalises from this, and what should a reader not take from it?

IC: What travels is the habit, more than the numbers. Measure what the agent writes before anything checks it. Keep a check that never changes, so you have a before to compare with. And look at what your supervision is made of as well as how much of it there is. I think the direction holds too: a rule a program can check shouldn’t be left to the agent’s memory, and a new model can do as much as your best rule and guard did for a single rule.

What shouldn’t travel are my percentages. I’m one person, with one tool, working mostly in Spanish, and I only wrote a rule after an agent had broken it again and again, so part of what looks like improvement may just be the bad days passing. And nobody should read this as proof that blocking an agent teaches it something. In my logs it didn’t show: the guard cleaned up what reached me, and the drafts after a block weren’t detectably better.

Putting the thesis to the test

Q9. What would falsify your argument — what result would make you abandon the guards approach?

IC: If a plain written rule, tested properly, did as well as a guard, I’d drop the guard for that rule and keep it as insurance at most. And if a guard stopped making what reaches me cleaner, or blocked more good work than bad, I’d switch it off myself. Its whole case is that it filters: after my writing guard went in, 11.2% of first drafts broke a rule but only 6.1% of the answers that reached me did.

My own data already push that way for writing. With the newest model, in the last days of the record, about one first draft in thirty broke one of my rules, and I think a filter with almost nothing to catch is close to retirement. Where I can’t imagine giving it up is on actions you can’t undo. I can’t prove that with my data. It’s a judgement, and I’d rather say so than dress it up as a finding.

Strongest Argument Against Guards

Q10. What is the strongest argument against guards — that they are just tests by another name, or scaffolding that better models will make obsolete?

IC: The second one, and my own data hand it part of the argument. A single model release did a large part of what my best combination of rule and guard did for the one rule I could follow. For rules about style, I fully expect better models to make guards pointless, and I’ll be happy to delete them.

But my busiest guard over the whole record wasn’t the one about writing. It was the one that stops destructive or forbidden commands, with 1,018 of the 3,411 blocks. A better model may try those less often, though I haven’t measured it, and rarer isn’t enough when the action is final. A deleted database costs the same whichever model deleted it.

Tests by another name? Partly. A test checks code once, before it ships. A guard checks behaviour every time, at the moment of acting, because an agent won’t necessarily do the same thing twice. It’s closer to what Ulysses did when he had himself tied to the mast: he knew he couldn’t trust himself at the one moment that mattered, so he decided beforehand. A guard is a decision I made in a calmer moment, enforced when I’m not looking.

First Guard Worth Writing 

Q11. If a team wanted to start tomorrow, what is the first guard worth writing?

IC: The one on the action you’d least like to explain afterwards. For most teams that’s something final: deleting data, overwriting someone’s work, sending a message to a customer, moving money. Write it so it checks the command before it runs and tells the agent in one line why it stopped. Decide beforehand what happens if the guard itself breaks. Mine step aside, because a guard that blocks good work gets switched off, but where money moves I’d rather it wait. Then feed it a bad command on purpose to see it bite, and have someone read what it blocked every week.

The second one I’d write stops the agent from saying a task is done when nothing shows it was checked. It was among my busiest, and I think it’s the cheapest one to write.


AI agent supervision interview: Hellomatik's Isaac Correa

TechRecast Analysis (The Recast)

The Shift in Context — Agent tooling has largely been built around instruction: system prompts, rules files, conventions documents. The implicit model is that an agent reads the rules and complies. A study of one operator’s logs suggests that model is weaker than it looks, and that enforcement — not persuasion — is what changes behaviour. That is where this AI agent supervision interview sits.

Why This Interview Matters — The material is unusually honest for this space. The author reports a null result for instructions, admits a confound in his own strongest before-and-after comparison, and states the study’s limits without being asked. That combination is rarer, and more useful, than a vendor benchmark — even though it is one operator and one toolset, and not peer reviewed.

Technology Perspective — The technical crux is placement. An instruction lives in the context window and competes with everything else there; a guard sits outside the model, between the intended action and the execution, and can refuse. That is a categorically stronger position. The cost is that guards must be written per rule and maintained, which is exactly the work instructions appeared to save.

Business and Market Perspective — The most commercially consequential finding is the model swap: if a vendor changes the underlying model and the rate of rule-breaking moves from 57% to 18% — or the reverse — then a product’s behaviour is partly outside the buyer’s control and invisible in a contract. That reframes procurement from feature comparison to change management.

Leadership Perspective

The author’s position — a non-programmer running many sessions at once — is the profile of the manager the agent era is creating, rather than the engineer it was built for. The relevant lens is whether his guard approach is reproducible by someone without his patience, or whether it depends on the specific operator.

The Counterpoint — A fair sceptic would argue that a single-operator study cannot support claims about supervision in general; that the guard-versus-wording comparison is confounded by the author’s own admission; that guards are regression tests renamed; and that as models improve the need for both instructions and guards declines. On this view, the paper is a useful field note, not evidence. This is the strongest opposing case and deserves a direct answer.

What Comes Next — Watch three signals: whether “guard” becomes a named pattern in agent tooling, whether procurement contracts start addressing model changes, and whether anyone replicates the instruction-versus-guard comparison with a design that separates the two.

Industry Outlook — If the finding holds, agent reliability work migrates from prompt engineering to enforcement, and the tooling market shifts accordingly. If it does not, instructions improve as models do, and guards remain a niche practice for high-stakes automation.

Final Note

Isaac Correa is co-founder of Hellomatik, a Madrid company that builds software for companies’ most complex work, using AI agents and apps that run on each company’s own data. The study discussed here is a self-published working paper, not peer-reviewed: “What changes a coding agent’s behaviour” (DOI 10.5281/zenodo.23510601, isaac-correa.github.io/coding-agents). ORCID 0009-0003-0657-4419