TL;DR: Vector search retrieval-augmented generation answers "what text resembles this query" — one similarity computation per chunk, independent of every other chunk. GraphRAG answers "how are these entities connected," at roughly 100–1000x the indexing cost. That cost is justified only when a measurable share of real query traffic is multi-hop or corpus-wide — not because knowledge graphs are structurally better retrieval, but because they're solving a different problem that vector similarity cannot solve at all, regardless of how well it's tuned.
RAG has largely settled on a default architecture: chunk documents, embed the chunks, retrieve the nearest neighbors to a query embedding, stuff them into a context window. It works well for a specific and common class of question — "what does the remote work policy say about equipment stipends" — where the answer lives in one or two chunks and the retrieval problem really is "find text that resembles this query."
It works poorly for a different class of question that shows up constantly in production once a corpus grows: "how did this vendor's role in the project change between Q1 and Q3," or "what's the overall theme of complaints across support tickets this quarter." These aren't lookups. They require connecting facts scattered across dozens or hundreds of documents, and vector similarity search has no mechanism for that — each chunk is scored against the query independently, with zero awareness of any other chunk. This is the actual argument for GraphRAG, and it's worth being precise about it, because "knowledge graphs are more accurate" is not really true in general — they're accurate at a specific thing vector search cannot do at all.
What vector RAG structurally cannot do
Vector similarity retrieval treats every chunk as an isolated unit. A query embedding gets compared against every chunk embedding, and the top-k most similar chunks get returned — full stop. There's no step where the system reasons about relationships between the returned chunks, or notices that chunk A mentions an entity that chunk B also discusses in a different context.
This produces what's been called "tunnel vision": retrieval that's locally correct (each chunk really is semantically close to the query) but globally blind. Two documented failure patterns:
- Multi-hop questions, where the answer requires chaining facts across documents ("who reports to the person who approved this vendor contract") — vector search can retrieve documents mentioning the vendor and documents mentioning approval chains, but has no mechanism to join them into a reasoning chain. Naive RAG has been benchmarked at roughly 40% accuracy on genuinely multi-hop questions, well below plan-based or graph-based approaches.
- Corpus-wide synthesis questions ("what are the recurring themes across all incident postmortems this year") — there's no unit of retrieval that represents "the whole dataset," only individual chunks, so the system can at best sample a handful of chunks and hope they're representative.
Neither failure is a tuning problem. Better chunking, better embeddings, and query expansion all help around the margins, but the fundamental limitation — independent per-chunk scoring with no cross-chunk reasoning — doesn't go away with a better embedding model. It's also worth noting that the retrieval layer is a different concern from securing LLM applications against prompt injection or untrusted content in retrieved documents — retrieval architecture and retrieval-content security are separate problems that both need solving.
What GraphRAG actually builds
Microsoft's GraphRAG (the reference implementation most production systems are compared against) runs a multi-stage indexing pipeline, and understanding the stages matters because each one is where the cost comes from:
- Text-unit chunking — same starting point as vector RAG.
- LLM-driven entity and relationship extraction — an LLM reads each chunk and extracts entities (people, organizations, concepts) and the relationships between them, using a domain-tunable entity-type list. This is the expensive step: it's an LLM call per chunk, not an embedding call.
- Entity and relationship summarization — descriptions of the same entity mentioned across many chunks get merged by another LLM pass into one coherent summary per entity, and likewise for relationships.
- Community detection — the resulting graph is partitioned using Leiden community detection, an algorithm that finds tightly interconnected clusters of entities, recursively, to build a hierarchy from fine-grained topics up to broad themes.
- Community summarization — an LLM summarizes each community into a report, so a "global" query can retrieve high-level synthesized context instead of individual graph nodes.
At query time, this supports two distinct retrieval modes: local search, which pulls a specific entity's neighborhood in the graph (good for "tell me about X and what it's connected to"), and global search, which pulls from the pre-generated community summaries to answer broad, dataset-wide questions — the thing vector RAG structurally cannot do.
Vector RAG: query → embed → nearest-neighbor chunks → LLM
GraphRAG: corpus → extract entities/relations → detect communities →
summarize communities → [query → local or global search] → LLMThe cost difference is not small
This is the part that gets understated in comparison posts. Microsoft Research's own LazyGraphRAG benchmark measured GraphRAG's original indexing pipeline at roughly $1,544 per million source tokens, against roughly $1.45 per million tokens for a vector embedding index — on the order of 1,000x more expensive to build, because the indexing cost scales with LLM inference work (entity extraction, summarization, community reports), not with disk size or embedding throughput.
Production cost breakdowns for a corpus large enough to produce ~10,000 communities put total indexing cost in the tens to hundreds of thousands of dollars depending on model choice (cheaper models for extraction meaningfully reduce this, and several teams report cutting GraphRAG token costs by 90% through selective extraction and cheaper summarization models — but even the optimized number is well above a vector index's cost).
And indexing isn't a one-time cost the way it can feel like with a vector store: as the corpus updates, entity and relationship extraction, community re-detection, and re-summarization all need to run again for affected regions of the graph, which is a meaningfully heavier update path than re-embedding new or changed chunks.
Decision framework: is the multi-hop share worth it
The question worth actually measuring, not guessing at, is: what fraction of real query traffic is asking something a similarity search cannot answer even in principle? If it's a small minority, GraphRAG's indexing cost and maintenance burden won't pay for itself — vector RAG with better chunking and maybe a reranking step will outperform it on cost-adjusted accuracy. If a meaningful share of queries are genuinely relational or corpus-wide, GraphRAG (or a hybrid) becomes the right call.
| Query pattern | Vector RAG | GraphRAG |
|---|---|---|
| "What does policy X say about Y" (single-document fact lookup) | Strong fit | Overkill |
| "How do entities A and B relate, possibly indirectly" (multi-hop) | Structurally weak | Strong fit |
| "What are the themes/trends across the whole corpus" (global synthesis) | Cannot do this | Strong fit (global search) |
| Fast-changing corpus, frequent updates | Cheap to keep current | Expensive to keep current |
| Small-to-medium corpus, budget-constrained | Right default | Hard to justify |
| Explainability requirement ("show me the reasoning path") | Weak — chunks aren't a reasoning trace | Strong — graph traversal is inspectable |
The pragmatic middle: hybrid retrieval
Most production systems that adopt GraphRAG don't replace vector search — they run both. Vector search handles the high-volume, cheap, single-hop majority of queries; graph retrieval is invoked selectively, either via query classification (route multi-hop-looking queries to graph search) or as a second-stage enrichment step that adds connected context around chunks vector search already found. This captures most of GraphRAG's benefit on the queries that actually need it, without paying full indexing and maintenance cost across a corpus where most queries don't need graph structure at all.
Choosing between retrieval architectures — and knowing which query patterns actually justify a knowledge graph — is a recurring conversation when we help clients build custom machine learning models and AI chatbots and assistants that need to reason over real, messy internal data rather than a clean benchmark corpus.
Sources
- GraphRAG vs Vector RAG: Which Retrieval Method is Best? — Analytics Vidhya
- Knowledge graph vs. vector RAG: Benchmarking — Neo4j
- Methods — Microsoft GraphRAG docs
- GraphRAG: New tool for complex data discovery — Microsoft Research
- How to improve multi-hop reasoning with knowledge graphs and LLMs — Neo4j
- Vector Search Is Not All You Need — Towards Data Science
- Cutting GraphRAG Token Costs by 90% in Production — Medium
- Reduce GraphRAG Indexing Costs: Optimized Strategies — FalkorDB