Ask AI the same thing two ways and you get two answers. Here's how a checking layer keeps the result consistent, whichever model you use.
Ask AI the same question two different ways and you’ll often get two different answers. Most people who have used it for real work have seen this. You phrase a request one way on Monday and another way on Tuesday, and what comes back doesn’t match. It’s the main reason business owners try AI, get uneasy about it, and quietly go back to doing the job by hand. That inconsistency isn’t a bug someone forgot to fix. These models work by predicting what comes next, so a slightly different question genuinely does produce a slightly different answer. Frank Coyle, who has spent decades in computer science and now teaches at Berkeley, makes the point that the imagination people call hallucination is a feature of how the technology works rather than a fault in it.
Which means the fix isn’t a better prompt or a newer model. The fix is to stop asking the model to be reliable and put a check around it instead. That’s the piece most businesses are missing, and it’s what makes agents safe enough to run across a whole organisation rather than in one careful corner of it.
Why the same request never arrives the same way
Picture a workwear supplier with an email or chat ordering process on its website. Trade customers message all day. One writes “6 x hi-vis shirts, XL, deliver Friday”. Another writes “hey mate, need about half a dozen of those orange shirts, biggish, sometime this week”. Both people want the same thing, and when its a person behind the counter, they can read both messages and know exactly what to do. An AI reading those two messages will produce two different results, and neither is guaranteed to arrive in the shape the ordering system needs. One comes back with a size of “XL” and the other with “biggish”. One sets a real delivery date and the other invents something close enough to look right. Nothing has gone wrong inside the model, it did what it is supposed to do, which is produce a likely answer. The problem is something software development has been grappling with for decades, how do you put guardrails around the requests a human is going to make in an free text scenario?
It helps to know what the model can and can’t do here. As Coyle puts it, an LLM can’t actually do anything. It can’t place the order. All it can do is read the message, work out what details belong in an order, and hand back a request that your ordering tool be called with those details. Your system does the placing. That’s good news, because it means there’s a gap between what the AI proposes and what your business actually does, and you can put your own checks in that gap.
A form at the door and a rulebook at the ledger
Coyle’s shorthand for this is worth borrowing. Pydantic, or a Type Validator at the door, ontology at the ledger. Two checks, doing different jobs.
The first is a form. Before anything reaches your ordering system, the AI’s output has to fit a fixed shape. Item code from your catalogue. Size from your real size list. Quantity as a whole number. Delivery date as a date. Customer account that exists. Developers use a tool called a Type Validator to describe that shape, and it means nothing gets through unless every field fits. “Biggish” fails at the door, and the AI is sent back with a note saying what was wrong.
The second check is a rulebook, a written-down map of how your business understands its own world. What a customer is, what an order is, what a refund is, and how they connect. The technical name for it is an ontology, which sounds heavier than it is, and the shape of data behind it has been in ordinary use for years. A Wikipedia search runs on one. What the rulebook catches is the stuff that reads perfectly well in English and is still wrong. A second refund on an order that’s already been refunded. A payout sent to the support rep who handled the case instead of the buyer, because the rulebook knows those are two different kinds of person. An order status of “probably shipped” when the only real statuses are paid, shipped, and refunded. A model will produce all three of those quite happily. Your rulebook doesn’t read English, so it tests each claim against your own definitions and rejects what doesn’t fit.
The same result whichever model you use
Because both checks sit outside the model, they don’t care which model produced the answer. The same form and the same rulebook apply to Claude, Gemini, ChatGPT, and Grok alike. You can swap models next year, or run a cheaper, open-weights one for simple messages and a stronger one for messy ones, and what reaches your systems keeps the same shape. Given how quickly these models get replaced, that’s worth more than picking a winner today.
Back at the workwear supplier, both messages end up as the same clean order. The vague one takes an extra round trip, where the AI asks which size the customer means, gets “XL” back, and passes. Something like 90% of chat requests could go straight through without a person touching them, with the rest landing in a queue for staff. Be plain about what this does and doesn’t guarantee. Two models won’t word a reply identically. What the checks guarantee is that whatever reaches your business is valid, or it doesn’t reach your business at all.
What this means for you
The check catches the wrong shape, not the wrong judgement. An order that’s correctly formatted and still a bad idea will sail through, so a person stays accountable for anything unusual, and it’s worth capping how many times an agent may retry before a human sees it, because every attempt costs money. The harder truth is that writing the rulebook is the real work. Most businesses have never written their own definitions down, and two people in the same company will often mean different things by “customer”. An agent won’t fix that. It will apply your muddle faster and in more places. Working out what you actually mean is the part that pays, and it’s the same work whether you have ten staff or a thousand, just at a different size.
That’s why the honest starting point is small. Pick one flow where messages already arrive in plain English and end in a structured action, write down the handful of things that flow touches, and see what comes out the other side. If you want to see how this is built in practice, our AI agents work covers the checking layer as well as the model, and there’s more on how agentic AI behaves in a real business on our technology pages.
See the full presentation from Frank Coyle on YouTube
