The Hidden Complexity of RAG — From Beginner Surface to Builder Depth
There is a version of RAG that takes two hours to build. Embed your documents. Store them in Chroma. Embed the query. Retrieve top-5 chunks. Pass to the LLM. Ship it. You will get a demo that impresses everyone in the room.
Then you put it in production. Users ask slightly different questions than you expected. The retrieval returns irrelevant chunks. The LLM hallucinates because it got noisy context. Sensitive customer data leaks into wrong responses. Latency spikes because you are embedding every query cold. Nobody told you any of this was coming, because every tutorial stopped at the demo.
This post is the complete iceberg map — above the waterline and below it.
- ›The beginner RAG stack (LangChain, embeddings, vector DB, prompt templates) is real and necessary. It is not wrong — it is just incomplete for production.
- ›Poorly evaluated RAG systems hallucinate in up to 40% of responses even when the correct source document was retrieved. The problem is not always retrieval — it is generation quality on noisy context.
- ›70% of RAG systems in production still lack systematic evaluation frameworks, making it impossible to detect quality regressions.
- ›The builder's layer — reranking, query reformulation, PII masking, hallucination detection — adds significant engineering effort but prevents the silent failures that break production systems.
Above the Waterline: RAG for Beginners
These are the 7 concepts every RAG tutorial covers. They are the foundation. Without them, nothing else works. The mistake is not learning them — the mistake is thinking they are enough.
Basic RAG Pipeline — the beginner's version
The demo problem: This pipeline works well on the documents you indexed and the queries you tested. It breaks on edge cases, domain-specific vocabulary, PII-containing documents, ambiguous queries, and any user behavior you did not anticipate. The above is not wrong — it is just incomplete without everything below the waterline.
Below the Waterline: RAG for Builders
This is where production systems diverge from demos. Each concept below exists because something broke — in someone's production system — before they added it. These are not optional improvements. They are the lessons of shipping RAG at scale.
Tier 1 — Pre-Retrieval: Before You Even Search
# Query reformulation with HyDE — Hypothetical Document Embeddings
from anthropic import Anthropic
client = Anthropic()
def hyde_retrieve(user_query: str, vectorstore) -> list:
"""Generate a hypothetical answer, embed it, use that for retrieval."""
hypothetical_response = client.messages.create(
model="claude-sonnet-4-6", max_tokens=256,
messages=[{
"role": "user",
"content": f"""Write a 2-paragraph document that would ideally answer
this question. Be specific and use domain vocabulary.
Question: {user_query}"""
}]
).content[0].text
# The hypothetical doc uses domain vocabulary the real docs also use
# Its embedding lands closer to relevant chunks than the sparse query
results = vectorstore.similarity_search(hypothetical_response, k=5)
return results
# Example: query "what happens if I miss a payment"
# HyDE generates: "If a customer fails to make a scheduled payment, the
# contract specifies a grace period of 5 business days after which..."
# That document embeds near the actual contract terms — better retrievalTier 2 — Post-Retrieval: After You Find Candidates
# Cross-encoder reranking — the accuracy boost that costs latency
from sentence_transformers import CrossEncoder
# Stage 1: Bi-encoder retrieves candidates fast (millions of docs, <50ms)
candidates = vectorstore.similarity_search(query, k=50)
# Stage 2: Cross-encoder re-scores top-50 candidates slowly but accurately
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
pairs = [[query, doc.page_content] for doc in candidates]
scores = reranker.predict(pairs) # (query, doc) processed together
# Sort by reranker score, take top-5 for LLM context
ranked = sorted(zip(scores, candidates), reverse=True)
top_docs = [doc for _, doc in ranked[:5]]
# Why this matters:
# Bi-encoder: "cat" and "dog" have similar embeddings -> both retrieved
# Cross-encoder: sees full (query + doc) pair -> scores by actual relevance
# Cost: 50 cross-encoder inference calls vs 1 ANN searchTier 3 — Generation: Quality, Safety, and Control
# Faithfulness evaluation with RAGAS
from ragas.metrics import faithfulness, context_precision, answer_relevancy
from ragas import evaluate
from datasets import Dataset
# Your RAG pipeline output
data = {
"question": ["What is the penalty for late payment?"],
"answer": ["There is a 5% late fee after 30 days."],
"contexts": [[
"Section 4.2: Late payment fees of 5% apply after 30 calendar days.",
"Section 4.3: Disputed invoices are exempt from late fees."
]],
"ground_truth": ["A 5% late fee applies after 30 calendar days."]
}
result = evaluate(Dataset.from_dict(data),
metrics=[faithfulness, context_precision, answer_relevancy])
print(result)
# faithfulness: 1.0 -> every claim is grounded in retrieved context
# context_precision: 0.8 -> 80% of retrieved chunks were actually needed
# answer_relevancy: 0.95 -> answer directly addresses the question
# Action thresholds (production):
# faithfulness < 0.8 -> investigate hallucination sources
# context_precision < 0.5 -> retrieval is surfacing too much noise
# answer_relevancy < 0.7 -> generation is going off-topicThe Deepest Layer — Rarely in Tutorials
The 5 Evaluation Metrics That Define Production Quality
Every metric below is measurable, automatable, and tells you a different thing about where your system is failing. You need all five — they are complementary, not redundant.
| Metric | What it measures | Target | What low score means |
|---|---|---|---|
| Faithfulness | Are answer claims supported by retrieved context? | ≥ 0.80 (≥0.90 regulated) | LLM is hallucinating from training memory |
| Context Precision | What fraction of retrieved chunks were actually needed? | ≥ 0.75 | Retrieval returning noisy irrelevant chunks → confuses LLM |
| Context Recall | Did retrieval find all relevant information? | ≥ 0.80 | Important information exists in corpus but was not retrieved |
| Answer Relevancy | Does the answer directly address the question? | ≥ 0.75 | Retrieval pulled adjacent-but-wrong chunks; generation is tangential |
| Hallucination Rate | % of responses with claims not in retrieved context | ≤ 5% for most apps | 5% = 1 in 20 responses contains fabricated information |
The teams I see building reliable production RAG are the ones who set up RAGAS evaluations before they set up Pinecone. They treat evaluation as the foundation, not as the polish at the end. Every builder-layer concept in this post exists because someone eventually measured what was going wrong — and only then could they fix it.
Three Things to Take Away
Three Things to Take Away
The beginner stack is not wrong — it is incomplete. LangChain, embeddings, vector databases, and prompt templates are the necessary foundation. Every builder-layer concept depends on having that foundation in place. The mistake is treating the foundation as the finished building.
Evaluation is the gateway to the builder layer. You cannot fix what you cannot measure. Context precision, context recall, faithfulness, and answer relevancy tell you specifically which part of your pipeline is failing. Without these metrics, improvements are guesswork. 70% of production RAG systems lack these — which is why 70% are stuck at demo quality.
The deepest layers are where competitive moats are built. Continuous fine-tuning on user feedback, semantic caching, multi-hop retrieval, secure tenant isolation — these are not features you add once. They are systems you build and maintain. The teams shipping reliable enterprise RAG at scale have invested engineering time in every layer of the iceberg, not just the visible tip.
The iceberg image that inspired this post captures it perfectly: what you see above water is a small fraction of what keeps the whole structure stable. The stability is entirely in what you cannot see.
Add RAGAS evaluation to your RAG pipeline this week. Run it on 50 queries from real user traffic. Check faithfulness, context precision, and answer relevancy. Post your scores — even if they are not impressive. Knowing your baseline is the first step to improving it, and most teams have no idea what their current scores are.