Half the market is arguing about whether RAG or a semantic layer is the right foundation for enterprise AI.
They are not competing. They answer different questions, and most teams need both.
Two shapes of question
Every question an agent receives breaks into one of two forms:
- “What did we say about X?” — lives in contracts, policies, tickets, docs. Unstructured. RAG was built for this.
- “What is true about X?” — lives in your warehouse and governed metrics. Structured. A semantic layer was built for this.
Treating them as rivals is how teams end up with a system that can quote the pricing policy but cannot tell you this quarter’s realised price.
Where each one breaks
| RAG | Semantic layer | |
|---|---|---|
| Good at | Retrieving relevant prose | Resolving definitions and joins |
| Fails on | Aggregation, math, current state | Anything not modelled as data |
| Permissions | Flattened at ingest, rebuilt at query time | Compiled per person, per query |
| Answer stability | Varies with retrieval ranking | Identical by construction |
| Audit story | Cites a chunk | Reproduces the exact SQL |
The permissions row is the one that ends pilots. A retrieval index that ingested everything has, by construction, assembled your most sensitive object — and reconstructing entitlement at query time is guesswork.
The layer that actually decides
Neither a document chunk nor a metric definition is worth much until something compiles it into a governed query and runs it.
That is the piece most architectures are missing: intent → context resolution → constrained planning → governed execution. RAG can feed the first step. It cannot perform the last three.
Point an agent at raw tables and the best models score in the low teens on real enterprise data. Give the same model compiled, governed context and it clears the high nineties. The retrieval quality was never the bottleneck.
The full breakdown — the precise division of labour, why hybrid architectures win, and how compile-time governance closes the gap RAG cannot — is here:
👉 RAG vs. Semantic Layer: Why AI Needs Deterministic Governance
Originally published at colrows.com/blogs/rag-vs-semantic-layer
