AI model reliability interview: how Meraki is Love’s Adam McClarin proves the setup beats the model
How much of an AI model’s reliability comes from the model, and how much from the setup around it? In this AI model reliability interview, Meraki is Love’s Adam McClarin reports a test across two frontier models where the setup, not the model, did most of the work — and explains how to test an AI assistant properly.
Meraki is Love LLC
Meraki is Love LLC builds AI assistants and web platforms for small businesses. Its work is not model development but integration: taking a frontier model and building the surrounding system — the standing context, the tool boundaries, the checks — that decides whether the assistant behaves reliably in production.
That distinction is the company’s whole argument. Most small businesses buy a model and expect reliability to come with it; Meraki is Love’s premise is that reliability is built around the model, and that the setup is where the engineering actually happens. It is the layer that is hardest to see in a demo and easiest to get wrong in a deployment.
Adam McClarin
Adam McClarin is the founder of Meraki is Love LLC. He is a CISSP with an MS in Cybersecurity, an Azure AI Engineer certification, and more than twenty years across software, security and AI — a security background that shapes the question he puts to AI systems.
His question is precise: how much of a model’s reliability comes from the model, and how much from the setup around it? He answers it with a test rather than an opinion. Running two frontier models through five tasks drawn from his own week, he found that setup beat the model — on a task with eight known failure patterns, one model hit all eight with nothing loaded, four when the rules were pasted into a single message, and none in two runs when the rules were standing context and the model could render and check its own output. This AI model reliability interview examines what that result actually proves.
McClarin is candid about the limits of his own test, including that the two setups were not equal and that both models invented customer reviews while one built a form that sent personal details nowhere. Readers will come away with a clearer view of why the model should never hold the scope, how server-side tenant enforcement closes a hole that query access opens, and why a passing test suite proves little until something has been broken on purpose.
Setup versus model
Q1. In one line, how much of a model’s reliability comes from the model, and how much from the setup around it?
AM: In my tests the setup moved the result more than the model did: the same Claude Opus 5.5 produced eight of eight design failures with nothing loaded and zero with standing context and a way to check its work.
Q2. Why does pasting rules into a single message perform worse than standing context with a render-and-check loop?
AM: A pasted list is read once, at the top of one request, and nothing afterward confirms the output followed it. When I typed my rules into a single message, Opus and Argon tied at four of eight failures, and Opus broke the first rule on the list, which was no italics. Inside Claude, Opus carried my saved preferences and the reasons behind them, and it could render its own page and scan the copy for italics, dashes and banned words before answering. That run reported two of its own mistakes and fixed them, and it produced zero failures, twice. Standing context tells the model what matters. The check loop catches the places where it drifted anyway.
The eight-failure-pattern test
Q3. Walk us through the eight failure patterns. What were they, and how did each setup score against them?
AM: The eight are the tells of what I call the Claude Effect, the look an AI hands back when nobody gives it direction. I scored each bakery home page against them: a serif headline with one italic accent phrase, a cream background with a terracotta accent, small all caps labels above headings, one word in the headline set in its own color, middle dots between labels, an arrow after every link, a grid of identical rounded cards, and every section fading up on scroll.
The scores, out of eight tells:
Opus with nothing loaded: 8.
Argon with nothing loaded: 4.
Opus with my rules typed into one message: 4, and it broke the italics rule.
Argon with my rules typed into one message: 4.
Opus in Claude with my saved rules, two runs: 0.
Argon in the Gemini app with my saved rules: 7.
Each of these is one run, and the setups were not identical, which my write-up says plainly. The numbers show a direction. They are not a benchmark.
Q4. Why did the second model not improve with saved rules — and what does that tell you about how to choose a model?
AM: Argon, with my rules saved in the Gemini app, produced seven of the eight tells, including Fraunces and italics, which my rules ban by name. With nothing loaded it produced four. So the saved rules did not hold, and the page got further from them.
That does not make Argon a weak model. It was the cheaper and faster one on the long build, about $0.67 against about $1.75, and it won two other tasks outright. The finding is that a public benchmark score says little about whether a model will follow your rules in your environment. Test the model on your own rules, in the setup you will actually run, and keep the first answer. Then choose on how it behaves there, with cost and speed as the second question.
Silent failure
Q5. Both models invented customer reviews, and one built a form that sent personal details nowhere. Why are these silent failures so hard to catch, and what would have caught them?
AM: Both pages looked finished. The reviews had names and warm sentences, and the form had labeled fields and a submit button. Nothing threw an error, and a visual check passes both. The failure lives in what the page does or claims, and a screenshot cannot show that.
Argon’s form collected a name, phone number and email and posted them to /api/v1/reservations, an endpoint that did not exist. The same page told visitors the server used parameterized SQL and an append-only audit table, for a server that did not exist either.
Three checks would have caught these. A content scan that flags any named testimonial and requires a source. A test that submits the form with a unique marker value and confirms the marker arrives in an actual datastore. And a read of the page against a short list of what it must not do, such as describe server internals to visitors.
Test for ‘Sent Nowhere’
Q6. How do you test for “sent nowhere” — a failure with no error and no visible symptom?
AM: I test the destination, since the page cannot tell me. Submit the form with a unique marker, something like a made-up name that appears nowhere else, then query the receiving store for that marker. If it is not there within a few seconds, the test fails. In the browser I also watch the network request for the submit and check the status code, because a post to a missing route returns a 404 that the page never shows.
For a form that is already live, I run the same marker submission on a schedule, so a deploy that breaks the route gets caught the same day and not weeks later when a client asks where their leads went. In the bakery case, I found the dead endpoint by reading the form action in the file, and a screenshot gave no hint.
No Tenant Field
Q7. You give the AI named, read-only functions with no tenant field, set server-side from the authenticated request. Walk us through why that closes a hole that query access opens.
AM: With query access, the model writes the query, and anything the model writes can be steered by whatever text it has read, including a message from an attacker. If the tenant filter lives in the query, the model can leave it out, or a crafted message can talk it out of the filter. Every customer’s data then sits one sentence away.
With named functions, a human reviews each function the way I would review any API endpoint. The model picks a function and supplies arguments, and the function’s schema has no tenant field. The server takes the tenant from the authenticated session and writes it into the WHERE clause before the query runs. The model has no argument it could use to ask for another tenant, so a prompt injection has nothing to grab.
My Meraki Inn assistant works this way at small scale. It has one tool, which takes an arrival date, a number of nights and a room. Its schema has no field about who is asking, and the server validates every value as if a stranger had posted it.
Designing a Tool Schema
Q8. What is the hardest part of designing a tool schema that the model cannot widen?
AM: Resisting the convenient field. Each time a feature is easier with one more parameter, such as an optional account ID, a free-text filter or a “search anything” string, the schema gets wider. A free-text field that ends up inside a query is the same hole as query access, only smaller.
The discipline is to list what the model genuinely needs to choose, which is usually a handful of enumerated values and validated dates or numbers, and to let the server supply everything else. Each argument then gets checked on arrival against a fixed list or range. In the inn assistant that means a room from the inn’s list, one to seven nights, a valid calendar date that is today or later and at most 540 days out. Anything else gets a plain reply asking again.
Removing the Scoping Clause
Q9. You remove the scoping clause on purpose and confirm the right tests fail. Why is a passing suite worthless until you have broken something deliberately?
AM: A green suite only proves that the tests passed. It does not prove the tests could fail. A test can pass because it checks the wrong thing, runs against the wrong target, or asserts something that is true whether the protection exists or not.
So I break the protection on purpose. I delete the tenant scoping clause, run the suite, and confirm the tests that should go red do go red. Then I restore the clause and confirm green. If nothing fails when the clause is gone, the suite was giving me comfort without evidence. I applied this to every fix in my own site review: each test was broken once and seen to fail before I trusted it.
Good Negative Test
Q10. What does a good negative test look like for an AI assistant?
AM: It proves the wrong answer is unavailable, and goes beyond showing that the right answer works. For a multi-tenant assistant, I ask for tenant B’s data while authenticated as tenant A, and I pick a question where tenant B’s record would be the best possible answer. The test passes only if the assistant returns nothing from B, even though B ranks first.
Other negative tests I run: put a card number in a message and confirm the request stops before it reaches the model. Send a conversation of absurd length and confirm it is refused before any spend. Send a request from another website’s origin and confirm it is turned away. Hand the model a tool call with a room or tenant it should not know, and confirm the server rejects it. For the file that touches the sensitive table, I add a chokepoint test that fails if any other file queries it.
What to insist on
Q11. If a small business is buying an AI assistant this month, what should it insist on, technically?
AM: 1. A fixed list of named, read-only functions in place of open query access, with no tenant or account field the model can fill in.
2. Spending ceilings at several levels: a rate limit at the edge, caps on message size, a cap on reply length, and a hard monthly limit on the AI account itself.
3. A written answer on where conversations are stored, for how long, how to delete them, and which providers see them, with each provider documented as a subprocessor.
4. Logs that record counts and outcomes and leave out the text of what customers typed.
5. Card numbers and similar sensitive input stopped before they reach the model.
6. Multi-factor sign-in on by default for every staff account, and an append-only audit log for security-relevant events.
7. Proof that the tests can fail, meaning the vendor can show a deliberate break turning the right tests red.
8. A statement of what the assistant cannot do, plus a hand-off to a person for everything the pages do not answer.
9. Written answers on backups and recovery, with no reliance on the assumption that the platform handles it.
If a vendor cannot show these, the business should ask why before signing.
Putting the thesis to the test
Q12. What would prove you wrong — is there a case where the model matters more than the setup?
AM: I would be wrong if one model, run inside the same setup as another, kept beating it by a wide margin. My test cannot rule that out yet. Each result is a single run, and the two apps I compared differed in more than the model, so the 8 to 0 gap mixes the model with the app around it.
There is one place in my data where the model did matter. In the security review task, both models got the same prompt in the same bare console, and both found all three planted flaws. Opus also explained why fixing the SQL alone leaves the account problem open, and it added row level security and a least privilege database user as backstops. Argon did not. No setup produced that difference in depth. The model did.
The test that would settle it is to run both models through identical standing rules and an identical render-and-check loop, ten times each, and score them against the same eight tells. If one still lands several tells ahead, the model matters more for that kind of work, and I would say so in print.
Setup Beats the Model
Q13. What is the strongest argument against “setup beats the model”, and how do you answer it?
AM: The strongest argument is that the two cannot be separated. A setup only helps if the model can use it. A standing rules file does nothing for a model that ignores it, and a render-and-check loop does nothing for a model that cannot read the result. My own data shows this: Argon with my rules saved produced seven of the eight tells, so the same kind of setup worked on one model and failed on the other. That suggests the model decides how much of the setup gets used.
I accept that. Where I hold my ground is on a narrower claim. The failures that cost a small business money, such as another customer’s data, a runaway AI bill, or a form that sends nothing anywhere, are closed by controls on the server that do not depend on the model behaving. The tenant comes from the authenticated session, the spending limit sits on the account, and the request is refused before the model ever sees it. Those controls hold with any model.
So the model sets the ceiling on how good the answers are, and the setup sets the floor on how badly things can go wrong. When a small business is choosing what to buy, I would check the floor first.
TechRecast Analysis (The Recast)
The Shift in Context
The AI conversation is still largely model-led: which model is best, which ranks highest, which is most capable. The counter-claim — that the setup around the model does most of the work — reframes reliability as an engineering problem rather than a procurement one. This AI model reliability interview sits on that reframing.
Why This Interview Matters
McClarin’s evidence is a small, honest experiment rather than a vendor claim: five tasks, two models, three setups, and a candid admission that the setups were not equal. That candour is what makes it worth interrogating — it is a test someone could reproduce, and it is rare in a field that mostly trades in benchmarks.
Technology Perspective
The technical crux is where authority lives. If the model can widen scope, no amount of prompt discipline fixes it; if authority is set server-side and the tool schema has no tenant field, the model cannot reach outside it. The scoping argument and the negative-testing argument are the same insight from two directions: design so failure is visible and bounded.
Business and Market Perspective
For small businesses, the practical consequence is that buying the best model is often the wrong first purchase. What they need is a builder who scopes tools, tests negative paths and designs the setup. That reframes what a small business should be buying — and what it should be asking.
Leadership Perspective
McClarin’s CISSP and twenty years in security are the relevant lens. Security engineers are trained to assume the component will fail and to design the boundary that contains it; applied to AI, that instinct produces exactly the scoping and negative-testing discipline he describes.
The Counterpoint
A fair sceptic would argue that the result is a single small test, that setups were not equal, and that model capability is improving fast enough to swamp setup differences. On this view, “setup beats the model” is true at the margin but overstated as a general law. This is the strongest opposing case and deserves a direct answer.
What Comes Next
Watch three signals: whether buyers start asking about tool scoping and negative tests before model choice, whether multi-tenant AI designs converge on server-side authority, and whether “break it on purpose” becomes standard practice in AI testing.
Industry Outlook
If McClarin is right, AI reliability work shifts from model selection to system design — schemas, boundaries, tests — and the model becomes a swappable component. If he is wrong, model improvements absorb the difference and setup engineering matters less each year.
Key Takeaways
1. Reliability comes substantially from the setup around a model, not the model alone.
2. In one test, standing context with a render-and-check loop beat rules pasted into a single message, which beat no setup at all.
3. The model should never hold the scope; tenant authority set server-side cannot be widened by anything the model outputs.
4. Silent failures — invented reviews, a form that sent data nowhere — are the ones demos miss and tests must target.
5. A passing suite proves little until you have broken something deliberately and seen the right tests fail.
6. The strongest counter-argument is that this is one small, unequal test, and model gains may swamp setup differences over time.