GraphRAG explained for CTOs: when a knowledge graph beats vector search
Plain RAG finds text that looks similar. GraphRAG finds facts that are connected. How they differ, when the graph is worth it, and how to test it in four weeks.
Most companies that put a language model in front of their documents built the same thing in 2024 and 2025: chunk the documents, embed the chunks, store the vectors, and at question time pull the most similar chunks into the prompt. This is retrieval augmented generation, RAG, and for a lot of questions it works.
Then someone asks a question like “which of our suppliers in Germany delivered parts for the products we recalled last year?” and the system answers confidently and wrongly. Nothing in the documents says that in one place. The answer lives in the relationships between suppliers, parts, products and recalls, and vector similarity cannot follow relationships.
That gap is what GraphRAG closes. This article explains what it is, when it is worth the extra work, and how to find out for your own data in a few weeks.
What plain RAG actually does
Vector retrieval answers one question: which pieces of text are most similar in meaning to the query? It is excellent at finding the paragraph that explains your refund policy or the ticket that describes a similar bug.
It struggles when the answer requires:
- Several hops. Supplier to part to product to recall is three joins. Similarity search returns chunks about suppliers and chunks about recalls and leaves the model to guess the connection.
- Aggregation. “How many”, “which ones”, “the total of” need the full set of matching facts, not the top ten most similar chunks.
- Precise identity. “Acme GmbH”, “ACME Germany” and “Acme” in three documents are the same supplier. Embeddings may or may not treat them as one.
- Structure that is not in text at all. Org charts, permissions, product hierarchies and bills of materials live in databases, not prose.
When the model does not have the connecting facts, it fills the gap from its training data or from plausible sounding text in the chunks. That is the hallucination your stakeholders complain about.
What GraphRAG adds
A knowledge graph stores entities (supplier, part, product, recall) and the typed relationships between them (supplies, contains, affected by). It can come from structured systems, from text with LLM based extraction, or from both.
GraphRAG uses this graph in retrieval. In practice there are three patterns and most real systems combine them:
- Graph traversal from entities in the question. Identify the entities mentioned, look them up in the graph, and walk their neighborhood. The model gets the supplier, its parts, the products and the recall as connected facts, with identifiers, not as loose paragraphs.
- Query generation. The model writes a structured query (SPARQL for RDF graphs, Cypher for property graphs) from the question. The database does the joins and the aggregation exactly. Text to query works well when the schema is clean and documented, which is a strong argument for an ontology.
- Community summaries. Clusters of related entities are summarized ahead of time, so global questions (“what are the main risk themes across our supplier base”) can be answered from summaries instead of thousands of chunks. This is the approach popularized by Microsoft’s GraphRAG paper.
Vector search does not disappear. It still finds the right text passages, and the graph provides the skeleton that tells the model how those passages relate. The retriever returns facts plus supporting text, and the model writes the answer with citations to both.
When the graph is worth it
Be honest about this part, because a knowledge graph is more work than an embedding index.
The graph pays off when your important questions have these properties:
| Property of your questions | Plain RAG | GraphRAG |
|---|---|---|
| Answer is in one or two passages | Good | No gain |
| Answer requires joining facts across documents or systems | Weak | Strong |
| Counting, listing all, aggregating | Weak | Strong |
| Entities appear under different names | Weak | Strong, with entity resolution |
| Answers must cite the exact source fact | Partial | Strong |
| Rules and permissions constrain what may be answered | Hard | Natural, the graph carries them |
| Domain is one product manual or FAQ | Good | Overkill |
If your questions are mostly in the first row, keep plain RAG and spend the budget on better chunking and evaluation. If they sit in the middle rows, you are paying for the gap in wrong answers and in people who stopped trusting the system.
What it costs
Three things make GraphRAG more expensive than plain RAG, and all three can be controlled.
Building the graph. Extracting entities and relationships from documents with an LLM costs tokens per document and needs a schema to extract against. Start with a small ontology: ten to twenty entity types and the relationships you actually query. Do not model the world.
Keeping it fresh. Documents change. The pipeline that built the graph has to run incrementally and validate what it adds. SHACL shapes or schema checks catch a model that suddenly starts extracting nonsense.
Query latency. A graph lookup plus vector search plus generation is slower than vector search plus generation. In our experience the difference is tens to a few hundred milliseconds, which users do not notice next to the model’s own generation time. Caching the graph neighborhood for common entities helps.
The cost that matters most is engineering time, and that is why the next section exists.
How to find out in four weeks instead of arguing
We run the same experiment with every client before recommending a graph.
Week one: the question set. Collect forty to sixty real questions from the people who will use the system, with the correct answers and where those answers come from. Include the hard multi hop ones. This set is the single most valuable artifact of the whole project and it outlives every model choice.
Week two: the baseline. Build or measure the plain RAG system on that question set. Score answer correctness and citation correctness. Write the numbers down.
Week three: a small graph. Extract a graph for the entity types that appear in the failing questions only. Use existing structured data where it exists, the database already knows which supplier delivered which part. Add graph retrieval to the pipeline.
Week four: compare. Same question set, same scoring. If the hard questions moved from wrong to right and the easy ones did not get worse, you have your answer and a working first version. If they did not move, you saved yourself months.
The pattern we see is that the graph lifts exactly the questions it was built for and leaves the others unchanged, which is what you want. It also produces a side effect clients did not ask for: the graph becomes a map of the data they have, with duplicates and gaps visible for the first time.
Practical advice if you go ahead
- Pick the graph model for your team, not for the paper. Property graphs (Neo4j and similar) are the faster start for most product teams. RDF with an OWL ontology is the better choice when you need standards, reasoning, or data exchange with partners and regulators. We build both and the decision rarely takes more than a conversation about your questions and your people.
- Make the schema visible to the model. Text to query works far better when the schema has clear names, descriptions and examples. This is ordinary documentation and it doubles as the ontology.
- Keep provenance. Every fact in the graph should know which document or system it came from. Citations become trivial and audits become possible.
- Evaluate in CI. The question set runs on every change to the prompt, the retriever or the graph. A regression shows up as a failed build, not as an email from a customer.
- Start with reads. Let the agent read the graph for months before it is allowed to write to it or act on the results. Permissions in the graph decide what the agent may see at all.
Summary
Vector search answers “what text is about this”. A knowledge graph answers “how are these things connected”. Questions that need the second kind of answer are exactly the ones plain RAG gets wrong, and they are usually the questions that matter most to the business. GraphRAG combines both. It costs more to build, the cost is controllable, and a four week experiment on your own questions tells you whether it is worth it.
If you want help running that experiment, our GraphRAG and AI agents service is built around it, and the knowledge graph development work that follows when the answer is yes.