An audit of 21 AI judge models shows simple agreement scores overstate AI quality. How Australian SMEs can check and govern the AI that makes their decisions.
A new audit looked at 21 AI judge models, the kind used to automatically grade the output of other AI systems, and found a problem that should worry anyone trusting AI to make calls. The common way of scoring these judges, simple agreement on how often the AI’s verdict matched a human’s, was heavily overstating how good they actually were. The researchers recommended stricter statistical measures, such as Cohen’s kappa, that account for agreement you would get by luck alone.
That sounds academic until you notice how many businesses now use AI as a judge. Grading support transcripts for quality. Reviewing contracts for risky clauses. Screening job applications. Scoring leads. In each case an AI is passing or failing something, and someone is acting on the verdict. If the score that says the AI is reliable is itself misleading, you have a flawed reviewer quietly approving bad work or rejecting good people, and a dashboard telling you everything is fine.
This piece is about that gap. Not the maths, but the practical question every small and mid-sized business should be able to answer about the AI it already uses. Can you actually trust the output, and how would you know.
Why “it mostly agrees” is not good enough
Here is the trap, in plain terms. Suppose you build an AI to flag support calls that need a manager’s attention, and in reality only one call in ten does. A lazy tool that flags nothing at all will still “agree” with the correct answer ninety per cent of the time, because ninety per cent of calls genuinely did not need attention. Ninety per cent sounds excellent. The tool is useless.
That is the heart of the audit’s finding. Raw agreement rewards the AI for getting the easy, common cases right, and the easy cases are usually the ones you did not need help with. The cases that matter are the rare ones, the bad transcript, the dodgy clause, the strong candidate with an unusual CV, and a high agreement score can hide complete failure on exactly those.
This is why the researchers reached for measures like Cohen’s kappa. The idea behind them is simple even if the formula is not. They ask how much the AI agrees with a human beyond what you would expect from guessing. A tool that scores well on that has earned it. A tool that scores well on raw agreement may just be riding the odds.
The lesson for an SME is not to go and learn statistics. It is to stop treating a single agreement percentage from a vendor as proof of anything. The right follow-up question is “agreement compared to what”, and if nobody can answer it, the number is decoration.
How to actually check an AI system
Checking an AI you depend on does not require a research team. It requires a method, run regularly.
Start with a hold-out set. Collect a sample of real examples from your own business, fifty to a few hundred depending on volume, and have a trusted person record the correct answer for each, before the AI sees them. This is your ground truth. Run the AI against it and compare. Because the answers were set independently, the comparison is honest in a way a vendor demo never is.
Look past raw agreement. Where you can, look at how the AI does on the cases that matter most: the ones it should have caught and the ones it wrongly flagged. A tool that is right ninety-five per cent of the time overall but misses half of the genuine problems is not safe for a job where missing a problem is the whole point.
Add human spot-checks. Even once a tool is live, pull a random sample of its decisions each week or month and have someone review them. This catches drift, where a tool that worked at launch slowly degrades as your inputs change, and it keeps a real person familiar with how the AI thinks.
Name an owner. Every AI making decisions in your business should have one person accountable for it, who knows what it does, what it is allowed to decide alone, and what gets escalated. When something goes wrong, and eventually it will, you want a name, not a shrug. This is the part most SMEs skip and most regret. Getting the testing and the accountability right is exactly where a technology partner earns their place, because the discipline matters more than the tool.
The Australian reality
Australia does not yet have a single binding AI law, but the direction is clear. The federal government’s AI Ethics Framework already sets out principles, fairness, transparency, accountability and contestability among them, and proposed mandatory guardrails are aimed squarely at high-risk uses. Screening people for jobs and making decisions about their money or access sit firmly in that category.
Fairness is the sharpest edge here. An AI that screens job applicants learns from past data, and if that data carried bias, the tool can repeat it at scale while looking neutral. Under the Privacy Act 1988 you are accountable for how personal information is used, and under anti-discrimination law you are accountable for the outcome, whether a human or a machine produced it. “The AI decided” is not a defence.
The safe pattern is the one regulators keep pointing to. Keep a human in the loop for consequential decisions. Let AI prepare, sort and recommend on the high-stakes work, hiring, rejecting, pricing, approving, but keep a person who can see the reasoning and overrule it making the final call. For the high-volume, low-risk work, lighter oversight with periodic audits is reasonable. Match the checking to the cost of being wrong.
What this means for you
The audit of those 21 judge models is a useful warning shot. The number that says your AI is trustworthy can be the least trustworthy thing in the room. Treat any single agreement score with suspicion, and ask what it is being compared against.
If you take three things from this, take these. Test every decision-making AI against your own hold-out set, not the vendor’s slides. Keep a human accountable, by name, for anything that affects a person’s job, money or rights. And review the tool on a schedule, because a system that was safe last quarter may not be safe now.
None of this is exotic. It is the ordinary discipline of responsible ai applied at a sensible size, and it is well within reach for a small business that decides to take it seriously. If you would rather not run that discipline yourself, our managed services and support work can keep the checks running so the AI stays honest after launch, not just on the day you bought it.
