AI agents stall on questions that run across several internal documents. Mapping how your data connects is what makes those answers reliable.
Ask an AI Agent a question about a particular topic and as long as the answer sits in one document, it usually does fine. Ask it something that runs through multiple documents and it starts guessing and cutting corners.
Say a supplier tells you an component batch was out of spec and needs to be recalled. Answering that correctly means working out which production runs used the batch, which products came out of those runs, which customers received them, and what your recall procedure says to do next. Each step depends on the answer before it. General AI search can’t follow that chain, because it isn’t really finding answers. It’s finding documents that look similar to your question, and similar-looking documents don’t add up to one connected answer.
That gap is what keeps AI agents from running on their own. An agent that comes back with something plausible but unrelated still needs a person to check it, and if a person has to check the very first step, you don’t have an agent, you have a bottleneck. The fix isn’t a smarter model, a better title for your agent, or a more expensive subscription. It’s giving the AI a map of how your information connects.
Why similar isn’t the same as connected
Ordinary AI search turns every document into a kind of numeric fingerprint, then finds the ones sitting closest to your question. That works well while the pile is modest and one document holds the answer. Add more, and more of them sit close together, so the results go blurry and the AI hands back things that are near your question without being about it.
Mapping the relationships fixes the blur. You write down what your business deals in and how those things connect. You get component details from your supplier. Those components go into these products, which went out on these dispatch notes to these customers. That map is called an ontology, and pairing it with ordinary AI search is called GraphRAG. The AI stops guessing which documents look related and starts following links you have told it are real.
Where the difference actually shows up
A benchmark study published this year compared plain AI search against graph-backed search across six sets of questions, using the same models and retrieval budgets throughout. On general questions, the ones where a single answer sits in a single document, the graph was worth about half a point on average. On multi-step questions, the kind where you have to find one thing in order to find the next, it was worth around 27 points. That difference is most of the buying decision. If your people mostly look things up, plain search is fine and cheaper to run. If they answer questions that thread through your records, the structure earns its keep.
Consistency matters more than the average when nobody is watching. The same study found graph-backed retrieval turned up the right supporting documents more often and varied far less from run to run. A separate study on legal judgments shows why that matters. With the correct documents in front of it, the model got 42.8% of charges right. Adding six plausible but irrelevant documents dropped that to 29.8%, and its accuracy on citing the right law fell from 74.7% to 46.8%. Poor retrieval doesn’t fail quietly. It walks a confident model into a wrong answer.
Those same researchers found something more uncomfortable. Models often produced the right answer without retrieving the evidence for it, which looks fine until somebody asks you to show your working. Their graph approach turned most of those into answers you could trace back to a source. For a manufacturer facing a recall question, that is the difference between an answer and one you can defend. Agents also get more done inside that structure. In a study of agents driving mobile apps, adding a graph lifted task completion from 46.9% to 75.8%.
The part nobody wants to do
None of this is new. Knowledge graphs have been around for decades. What stops businesses using them is that somebody has to describe how their information actually relates, and that work is unglamorous and specific to you. AI can help draft the map, but only your people know that the dispatch notes are the reliable record and the spreadsheet is three weeks behind. This is the ongoing side of Data Operations, not a project with an end date.
It changes as your business changes, which means a regular review (monthly is sensible), and version control so you can roll back an update that makes the answers worse. You will also want more than one map. Each department and team ask different kinds of questions depending on the job, so the map that serves a recall question isn’t the one that serves a costing question.
There is a bill attached too. Building these indexes takes processing time and money before anyone asks a question. In the benchmark, one graph method cost over $13 per million tokens of content and took about 1.7 hours to build, while another was effectively free. The effort also stops paying at some point. In the mobile agent study, mapping past about four hours per application produced no further gain, which tells you this is a job you can finish rather than feed forever.
Worth being straight about the other side. Graph structure isn’t the only way to make an agent search better. One recent study reached 68% accuracy on hard research questions with no graph at all, against 52% for a conventional search model twice its size, simply by feeding the agent’s own reasoning into the search step alongside its query. Stronger models also close part of the gap by themselves. Structure is the lever that pays when questions run across your own data, and what an agent is allowed to do with what it finds matters just as much, which is the job of the Agent Harnesses we build around them.
What this means for you
Before you get sold on adding AI agents, ask what they are going to do when a question crosses two systems. That one question separates a good demo from something that will survive contact with your business.
Then start with one job rather than the whole business. Write down the handful of things that job deals in and how they link, connect that map to the records you already keep, and let an agent work inside it. A person still owns the map and the judgement calls, because the map only reflects how your business actually works if your people say so. To see how we approach this kind of work, have a look around our website.
