Post

RAG Deep Dive: Chunking, Indexing, Hybrid Search, and Reranking in Production

Posted
Preview Image
By eya_garci 24 min read

Retrieval-Augmented Generation (RAG) is often described as “connect your LLM to your data.” That description understates the engineering problem. A production RAG system involves at least six distinct subsystems — each with its own failure modes, tuning parameters, and quality metrics. Getting one of them wrong can degrade the entire pipeline even if the model is state-of-the-art.

This article walks through each layer of a serious RAG system: document ingestion, chunking strategy, embedding, indexing, hybrid retrieval, reranking, and prompt injection. The goal is to give you the vocabulary and intuition to make good engineering decisions at each stage.

1. The RAG Architecture at a Glance

A RAG system sits between the user and the LLM. Its job is to find the most relevant content from a knowledge source and give that content to the model as grounded context.

flowchart LR
  A[User Query] --> B[Query Encoder]
  B --> C{Retriever}
  C --> D[Dense Search\nVector DB]
  C --> E[Sparse Search\nBM25 / ElasticSearch]
  D --> F[Fusion & Reranker]
  E --> F
  F --> G[Top-K Chunks]
  G --> H[Prompt Builder]
  H --> I[LLM]
  I --> J[Grounded Response]

The quality of the final response depends on every step. A powerful LLM cannot compensate for a poor retriever that returns irrelevant chunks.

2. Document Ingestion and Preprocessing

Before chunking, documents must be cleaned and normalized. Common issues in raw documents:

  • boilerplate headers, footers, and watermarks
  • non-semantic whitespace and formatting artifacts
  • tables and figures that do not parse cleanly as text
  • HTML or PDF extraction errors

A minimal preprocessing pipeline should:

  1. extract text faithfully (PyMuPDF for PDFs, BeautifulSoup for HTML)
  2. strip boilerplate using heuristics (repeated short lines, page numbers)
  3. normalize whitespace
  4. preserve section structure metadata (title, section, page) as document-level tags

Metadata is critical. Every chunk you store should carry the document ID, section title, page number, and source URL. You will use these for citation and for filtering during retrieval.

3. Chunking Strategies

Chunking determines the unit of retrieval. The choice has a direct impact on retrieval precision and context coherence.

Fixed-length with overlap

Split every N tokens with an M-token overlap. Simple and fast. Works well for uniform prose where logical units are hard to detect automatically.

1
2
3
4
5
6
7
8

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=64,
    length_function=len,
)
chunks = splitter.split_text(document_text)

The overlap prevents splitting a concept across two chunks with no shared context.

Structure-aware chunking

Split on logical boundaries: headings, paragraphs, list items, or code blocks. Requires document structure information (Markdown, HTML, or PDF structure extraction).

This is usually better than fixed-length for technical documentation, because it preserves the semantic unit the author intended.

Semantic chunking

Use an embedding model to detect topic boundaries. Measure cosine similarity between consecutive sentence embeddings. Split when similarity drops below a threshold.

1
2
3
4
5
6
7
8
9

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

chunker = SemanticChunker(
    embeddings=OpenAIEmbeddings(),
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=95,
)
chunks = chunker.split_text(document_text)

Semantic chunking is more expensive (requires embedding during ingestion) but produces more coherent chunks. Recommended for heterogeneous corpora.

Hierarchical (parent-child) chunking

Store both small chunks (for precise retrieval) and larger parent sections (for coherent context). Retrieve the small chunk to determine relevance, but inject the parent section into the prompt.

1
2
3
4
5

Document
  └── Section (parent chunk, ~2000 tokens)
        ├── Paragraph A (child chunk, ~200 tokens)
        ├── Paragraph B (child chunk, ~200 tokens)
        └── Paragraph C (child chunk, ~200 tokens)

This gives you the precision of small chunks with the coherence of larger context windows.

Choosing a strategy

Use caseRecommended strategy
Uniform long-form proseFixed-length with overlap
Structured technical docsStructure-aware
Heterogeneous corpusSemantic
Complex reasoning tasksHierarchical (parent-child)
Code repositoriesFile-level + function-level

4. Embedding Models

The embedding model maps chunks and queries into a shared vector space. The choice matters enormously for retrieval quality.

Key dimensions to evaluate

  • Retrieval benchmark scores: MTEB leaderboard (Massive Text Embedding Benchmark) provides standardized scores.
  • Context window: some models support up to 8192 tokens; others max out at 512. Shorter windows are a problem for large chunks.
  • Dimensionality: higher-dimensional embeddings capture more information but cost more storage and compute.
  • Language coverage: if your corpus is multilingual, use a multilingual model (e.g., multilingual-e5-large).
  • Domain fit: general-purpose models (OpenAI text-embedding-3-large, Cohere embed-v3) are strong defaults; domain-specific fine-tuned models can outperform them on specialized corpora.

Asymmetric retrieval

Query and document should sometimes be encoded differently. Models like bge-large-en-v1.5 support instruction prefixes:

1
2
3
4
5

query_instruction = "Represent this sentence for searching relevant passages: "
doc_instruction = "Represent this passage for retrieval: "

query_embedding = embed(query_instruction + user_query)
doc_embedding = embed(doc_instruction + chunk_text)

This asymmetric encoding improves retrieval for information-seeking queries.

5. Indexing and the Vector Database

Once you have embeddings, you store them in a vector index that supports approximate nearest-neighbor search.

ANN index types

Index typeCharacteristics
Flat (exact)Perfect recall, O(n) scan — only for small datasets
IVF (Inverted File)Cluster-based partitioning, fast for large datasets
HNSW (Hierarchical NSW)Graph-based, very high recall, good insert speed
ScaNN / DiskANNOptimized for billion-scale datasets

Most production vector databases (Pinecone, Weaviate, Qdrant) default to HNSW. You rarely need to tune the index algorithm directly, but you should understand the recall vs. latency tradeoff.

Metadata filtering

Pre-filtering by metadata dramatically improves precision and reduces irrelevant retrieval:

1
2
3
4
5
6
7
8
9

results = collection.query(
    query_embeddings=[query_embedding],
    n_results=20,
    where={
        "source": "hr-policy",
        "year": {"$gte": 2023},
        "language": "en"
    }
)

Always store filterable metadata at index time. Retroactively adding filters to a large index is expensive.

6. Hybrid Search: Dense + Sparse

Pure vector search misses exact keyword matches. A user querying “CVE-2024-1234” or “Q3 2024 revenue” gets better results from a keyword-based lexical search engine than from dense retrieval alone.

Hybrid search combines both:

  • Dense retrieval: semantic similarity via embeddings (recall for paraphrases)
  • Sparse retrieval: BM25 or TF-IDF (precision for exact terms, rare words, codes)

Reciprocal Rank Fusion (RRF)

RRF is the standard fusion method. It does not require score normalization and is robust across retrieval systems:

\[\text{RRF}(d) = \sum_{r \in R} \frac{1}{k + r(d)}\]

where $r(d)$ is the rank of document $d$ in retrieval system $r$ and $k$ is a constant (typically 60).

1
2
3
4
5
6
7

def reciprocal_rank_fusion(dense_results, sparse_results, k=60):
    scores = {}
    for rank, doc_id in enumerate(dense_results):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    for rank, doc_id in enumerate(sparse_results):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

Most modern vector databases (OpenSearch, Weaviate, Qdrant) support hybrid search natively.

7. Reranking

After retrieval you typically have 20–50 candidates. A reranker scores each (query, chunk) pair more precisely than the embedding-based retrieval — but at higher cost.

Cross-encoder reranking

A cross-encoder takes the full (query, passage) pair and outputs a relevance score. It is much slower than bi-encoder retrieval but significantly more accurate.

1
2
3
4
5
6
7
8
9

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

pairs = [(query, chunk) for chunk in candidates]
scores = reranker.predict(pairs)

ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
top_k = [chunk for chunk, _ in ranked[:5]]

Cohere Rerank API

For a managed option:

1
2
3
4
5
6
7
8
9

import cohere

co = cohere.Client(api_key)
response = co.rerank(
    model="rerank-english-v3.0",
    query=user_query,
    documents=candidates,
    top_n=5,
)

When to use a reranker

  • Always use a reranker when precision matters more than latency.
  • Use a lightweight model (MiniLM-based) for latency-constrained paths.
  • Use a heavy model (Cohere, BGE-reranker-large) for offline or low-traffic critical tasks.

8. Query Transformation

Raw user queries are often noisy, ambiguous, or incomplete. Transforming the query before retrieval improves recall.

HyDE (Hypothetical Document Embeddings)

Generate a hypothetical answer to the query, then embed that answer for retrieval. The hypothesis is more likely to match document-style text than the original question.

1
2
3
4
5

def hyde_retrieve(query, llm, retriever):
    hypothetical_doc = llm.generate(
        f"Write a passage that would answer this question: {query}"
    )
    return retriever.retrieve(hypothetical_doc)

Multi-query retrieval

Generate N reformulations of the query, retrieve for each, and fuse results:

1
2
3
4
5

reformulations = llm.generate(
    f"Generate 3 different ways to ask this question:\n{query}"
)
all_results = [retriever.retrieve(q) for q in reformulations]
merged = reciprocal_rank_fusion(*all_results)

Step-back prompting

Rephrase the query at a higher level of abstraction. “What are the side effects of ibuprofen for pregnant women?” becomes “What are the general safety considerations for NSAIDs during pregnancy?” — retrieving broader context that contains the specific answer.

9. Context Injection and Prompt Construction

Retrieved chunks must be assembled into a prompt that gives the model enough context without overwhelming it.

Ordering matters

Position chunks with the most relevant content at the beginning and end of the context window. Research on “lost in the middle” shows LLMs attend less to content in the middle of long contexts.

Citation anchoring

Include chunk identifiers in the injected context and instruct the model to cite them:

1
2
3
4
5
6
7
8

You are an assistant answering questions based on the provided context.
For each claim, cite the source using [DOC-1], [DOC-2], etc.

Context:
[DOC-1] {chunk_1_text}
[DOC-2] {chunk_2_text}

Question: {user_query}

Faithfulness constraint

Explicitly instruct the model not to use knowledge outside the provided context:

1
2
3

Answer based ONLY on the context provided above.
If the answer is not in the context, respond with:
"I don't have enough information to answer this question."

10. Evaluating a RAG Pipeline

Evaluation must cover each subsystem independently.

Retrieval metrics

MetricWhat it measures
Recall@kFraction of relevant docs in top-k
Precision@kFraction of top-k that are relevant
MRRMean reciprocal rank of first relevant result
NDCGGraded relevance rank quality

Generation metrics (RAGAS)

MetricWhat it measures
FaithfulnessIs the answer grounded in retrieved context?
Answer relevanceDoes the answer address the question?
Context precisionAre the retrieved chunks relevant?
Context recallIs the relevant information retrieved?
1
2
3
4
5
6
7
8

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision

result = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy, context_precision],
)
print(result)

End-to-end evaluation

Use LLM-as-a-judge to score (question, context, answer) triples:

1
2
3
4
5
6
7
8
9
10
11

judge_prompt = """
Rate the answer on a scale of 1-5:
1 = completely wrong or hallucinated
5 = fully correct, grounded in context, and complete

Question: {question}
Context: {context}
Answer: {answer}

Score:
"""

11. Common Failure Patterns and Fixes

FailureSymptomFix
Chunk boundary cuts key infoAnswer misses part of the relevant passageAdd overlap, use structure-aware chunking
Retriever returns irrelevant chunksLLM hallucinates or says “I don’t know”Add reranker, improve embedding model, add metadata filters
Exact term not retrieved by denseAcronyms, codes, rare terms missedAdd BM25 hybrid search
Context overflowLLM ignores part of the contextReduce chunk count, use hierarchical chunking
LLM ignores retrieved contextAnswers from parametric memoryStrengthen faithfulness constraint in system prompt
Slow retrieval latencyUser-facing p95 > 500msAdd ANN index, enable metadata pre-filtering, cache common queries

Putting It All Together

A production RAG system is an engineering product, not a model parameter. The model is one component. The chunking strategy, embedding model, index configuration, hybrid search fusion, reranker, and prompt template all contribute to the final answer quality and operational cost.

12. Corrective RAG (CRAG)

Corrective RAG adds a self-evaluation step after retrieval — if retrieved documents are judged irrelevant, the system queries a web search engine for supplementary context:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

from langchain_openai import ChatOpenAI
from langchain_community.tools import TavilySearchResults

llm     = ChatOpenAI(model="gpt-4o-mini", temperature=0)
web     = TavilySearchResults(max_results=3)

def evaluate_relevance(question: str, docs: list[dict]) -> str:
    """Returns 'relevant', 'ambiguous', or 'irrelevant'."""
    doc_texts = "\n\n".join(d["text"][:500] for d in docs)
    prompt = f"""
Does the following context adequately address the question?
Question: {question}
Context: {doc_texts}

Respond with one word: relevant, ambiguous, or irrelevant.
"""
    return llm.invoke(prompt).content.strip().lower()

def corrective_rag(question: str, retriever, llm) -> str:
    # Step 1: Initial retrieval
    docs      = retriever.retrieve(question, top_k=5)
    relevance = evaluate_relevance(question, docs)

    if relevance == "irrelevant":
        # Step 2a: Fall back to web search
        web_results = web.invoke(question)
        context     = "\n\n".join(r["content"] for r in web_results)
    elif relevance == "ambiguous":
        # Step 2b: Combine local + web
        web_results  = web.invoke(question)
        web_context  = "\n\n".join(r["content"] for r in web_results)
        local_context = "\n\n".join(d["text"] for d in docs)
        context      = local_context + "\n\n[Web search supplement]\n" + web_context
    else:
        context = "\n\n".join(d["text"] for d in docs)

    # Step 3: Generate answer
    return llm.invoke(f"Context:\n{context}\n\nQuestion: {question}\n\nAnswer:").content

13. Self-RAG

Self-RAG trains the model to decide when to retrieve, what to retrieve, and how to critique its own output using special reflection tokens:

1
2
3
4

[Retrieve]: Should I retrieve documents for this question?
[Relevant]: Is this retrieved document relevant?
[Supported]: Is this claim supported by the context?
[Utility]: Is this response useful?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

# Simplified Self-RAG inference loop
def self_rag(question: str, retriever, llm, max_steps: int = 5) -> str:
    context     = []
    draft       = ""
    final_answer = None

    for step in range(max_steps):
        # Prompt with Self-RAG tokens
        prompt = f"""
Question: {question}
Previous context: {chr(10).join(context)}
Current draft: {draft}

Should you retrieve more information? If yes, what query?
Format: retrieve
"""
        decision_text = llm.invoke(prompt).content
        import json
        try:
            decision = json.loads(decision_text)
        except json.JSONDecodeError:
            break

        if not decision.get("retrieve", False):
            # Model decided it has enough context
            final_answer = llm.invoke(
                f"Based on this context:\n{chr(10).join(context)}\n\nAnswer: {question}"
            ).content
            break

        # Retrieve
        query = decision.get("query", question)
        docs  = retriever.retrieve(query, top_k=3)
        context.extend([d["text"] for d in docs])

    return final_answer or draft

14. Agentic RAG with LangGraph

Production RAG systems often need multiple retrieval strategies, dynamic planning, and fallback behavior:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

from langgraph.graph import StateGraph, END
from typing import TypedDict, Annotated
from langgraph.graph.message import add_messages

class RAGState(TypedDict):
    question:       str
    retrieved_docs: list[dict]
    answer:         str | None
    retrieval_score: float
    retry_count:    int
    web_searched:   bool

def retrieve_node(state: RAGState) -> RAGState:
    docs = retriever.retrieve(state["question"], top_k=5)
    return {"retrieved_docs": docs}

def evaluate_retrieval(state: RAGState) -> RAGState:
    score = evaluate_retrieval_quality(state["question"], state["retrieved_docs"])
    return {"retrieval_score": score}

def grade_retrieval(state: RAGState) -> str:
    if state["retrieval_score"] >= 0.7:
        return "generate"
    if not state["web_searched"]:
        return "web_search"
    return "generate"    # use what we have, acknowledge uncertainty

def web_search_node(state: RAGState) -> RAGState:
    results = web_tool.invoke(state["question"])
    extra   = [{"text": r["content"], "source": r["url"]} for r in results]
    return {
        "retrieved_docs": state["retrieved_docs"] + extra,
        "web_searched":   True,
    }

def generate_node(state: RAGState) -> RAGState:
    context = "\n\n".join(d["text"] for d in state["retrieved_docs"][:5])
    answer  = llm.invoke(f"Context:\n{context}\n\nQuestion: {state['question']}\n\nAnswer:").content
    return {"answer": answer}

graph = StateGraph(RAGState)
graph.add_node("retrieve",        retrieve_node)
graph.add_node("evaluate",        evaluate_retrieval)
graph.add_node("web_search",      web_search_node)
graph.add_node("generate",        generate_node)
graph.set_entry_point("retrieve")
graph.add_edge("retrieve",   "evaluate")
graph.add_conditional_edges("evaluate", grade_retrieval, {
    "generate":   "generate",
    "web_search": "web_search",
})
graph.add_edge("web_search", "evaluate")
graph.add_edge("generate",   END)

app = graph.compile()
result = app.invoke({"question": "What is the latest LLM benchmark result?", "retry_count": 0, "web_searched": False})
print(result["answer"])

15. RAG Production Infrastructure

End-to-end document ingestion pipeline

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

from pathlib import Path
from langchain_community.document_loaders import (
    PyPDFLoader, UnstructuredMarkdownLoader, TextLoader
)
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
import hashlib

class ProductionRAGPipeline:
    def __init__(self, persist_dir: str = "./chroma_db"):
        self.embeddings  = OpenAIEmbeddings(model="text-embedding-3-large")
        self.vectorstore = Chroma(
            persist_directory=persist_dir,
            embedding_function=self.embeddings,
        )
        self.splitter = RecursiveCharacterTextSplitter(
            chunk_size=1000,
            chunk_overlap=100,
            separators=["\n## ", "\n### ", "\n\n", "\n", ". "],
        )
        self._indexed: set[str] = set()

    def _doc_hash(self, path: str) -> str:
        return hashlib.md5(Path(path).read_bytes()).hexdigest()

    def ingest(self, file_path: str, metadata: dict = None) -> int:
        doc_hash = self._doc_hash(file_path)
        if doc_hash in self._indexed:
            return 0   # already indexed, skip

        ext    = Path(file_path).suffix.lower()
        loader = {
            ".pdf": PyPDFLoader,
            ".md":  UnstructuredMarkdownLoader,
            ".txt": TextLoader,
        }.get(ext, TextLoader)(file_path)

        docs   = loader.load()
        chunks = self.splitter.split_documents(docs)

        for chunk in chunks:
            chunk.metadata.update({
                "source":   file_path,
                "doc_hash": doc_hash,
                **(metadata or {}),
            })

        self.vectorstore.add_documents(chunks)
        self._indexed.add(doc_hash)
        return len(chunks)

    def ingest_directory(self, directory: str, glob: str = "**/*.pdf") -> dict:
        stats = {"total_files": 0, "total_chunks": 0, "skipped": 0}
        for path in Path(directory).glob(glob):
            n = self.ingest(str(path))
            stats["total_files"]  += 1
            stats["total_chunks"] += n
            if n == 0:
                stats["skipped"] += 1
        return stats

    def query(self, question: str, top_k: int = 5, filters: dict = None) -> list:
        search_kwargs = {"k": top_k}
        if filters:
            search_kwargs["filter"] = filters
        return self.vectorstore.similarity_search_with_score(question, **search_kwargs)

16. RAG Observability and Monitoring

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

import time
import json
from dataclasses import dataclass

@dataclass
class RAGTrace:
    request_id:      str
    question:        str
    retrieval_ms:    float
    reranking_ms:    float
    generation_ms:   float
    total_ms:        float
    doc_count:       int
    top_doc_score:   float
    prompt_tokens:   int
    completion_tokens: int
    faithfulness:    float | None  # from LLM judge

def trace_rag_pipeline(question: str, retriever, llm, judge=None) -> dict:
    import uuid
    request_id = str(uuid.uuid4())
    t_start    = time.perf_counter()

    # Retrieval
    t0   = time.perf_counter()
    docs = retriever.retrieve(question, top_k=5)
    ret_ms = (time.perf_counter() - t0) * 1000

    # Generation
    context = "\n\n".join(d["text"] for d in docs)
    t1      = time.perf_counter()
    response = llm.invoke(f"Context:\n{context}\n\nQ: {question}")
    gen_ms  = (time.perf_counter() - t1) * 1000
    answer  = response.content

    # Optional faithfulness check
    faith = None
    if judge:
        faith = judge.check_faithfulness(context, answer)

    trace = RAGTrace(
        request_id=request_id,
        question=question[:100],
        retrieval_ms=round(ret_ms, 1),
        reranking_ms=0,
        generation_ms=round(gen_ms, 1),
        total_ms=round((time.perf_counter() - t_start) * 1000, 1),
        doc_count=len(docs),
        top_doc_score=docs[0]["score"] if docs else 0,
        prompt_tokens=response.usage.prompt_tokens,
        completion_tokens=response.usage.completion_tokens,
        faithfulness=faith,
    )

    # Log structured trace
    print(json.dumps({
        "request_id": trace.request_id,
        "retrieval_ms": trace.retrieval_ms,
        "generation_ms": trace.generation_ms,
        "total_ms": trace.total_ms,
        "doc_count": trace.doc_count,
        "faithfulness": trace.faithfulness,
    }))

    return {"answer": answer, "trace": trace}

RAG Optimization Roadmap

Start with a simple baseline and add complexity only where evaluation shows it’s needed:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

Phase 1 — Baseline (Day 1):
  Fixed-length chunks (512 tokens, 64 overlap)
  + Single embedding model (text-embedding-3-small)
  + Vector-only retrieval (k=5)
  + Simple prompt template

  Measure: RAGAS faithfulness, answer relevancy

Phase 2 — Retrieval Quality (Week 1):
  + BM25 hybrid search (0.3 BM25 : 0.7 dense)
  + Cross-encoder reranker (MiniLM)
  + Metadata filtering (date, source, department)

  Measure: Context precision, context recall, MRR

Phase 3 — Chunking (Week 2):
  + Structure-aware chunking (heading boundaries)
  + Parent-child indexing (retrieve small, inject large)
  + Semantic chunking for heterogeneous corpora

  Measure: Chunk boundary failure rate, faithfulness delta

Phase 4 — Query Enhancement (Week 3):
  + HyDE for low-overlap queries
  + Multi-query retrieval for ambiguous questions
  + Step-back prompting for abstract questions

  Measure: Recall@k improvement on hard queries

Phase 5 — Advanced (Month 2):
  + Corrective RAG (web search fallback)
  + Agentic RAG with LangGraph
  + Graph RAG for relational domains

  Measure: Full RAGAS suite, human eval on hard cases

RAG Evaluation Suite

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
    answer_correctness,
)
from datasets import Dataset
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper

class RAGEvaluationSuite:
    def __init__(self, rag_fn, judge_model: str = "gpt-4o-mini"):
        self.rag_fn       = rag_fn
        self.judge_llm    = LangchainLLMWrapper(ChatOpenAI(model=judge_model))
        self.judge_emb    = LangchainEmbeddingsWrapper(OpenAIEmbeddings())

    def run(self, test_cases: list[dict], baseline: dict = None) -> dict:
        results = []
        for case in test_cases:
            rag_output = self.rag_fn(case["question"])
            results.append({
                "question":     case["question"],
                "answer":       rag_output["answer"],
                "contexts":     rag_output["contexts"],
                "ground_truth": case.get("ground_truth", ""),
            })

        dataset = Dataset.from_list(results)
        scores  = evaluate(
            dataset,
            metrics=[faithfulness, answer_relevancy, context_precision, context_recall, answer_correctness],
            llm=self.judge_llm,
            embeddings=self.judge_emb,
        )
        result_dict = scores.to_pandas().mean(numeric_only=True).to_dict()

        if baseline:
            result_dict["deltas"] = {
                k: round(result_dict.get(k, 0) - baseline.get(k, 0), 4)
                for k in baseline
            }

        return result_dict

End-to-End Production RAG: Complete Code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_community.retrievers import BM25Retriever
from langchain.retrievers import EnsembleRetriever
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from sentence_transformers import CrossEncoder
import time, json

class ProductionRAG:
    def __init__(self, docs, model: str = "gpt-4o-mini"):
        self.llm       = ChatOpenAI(model=model, temperature=0)
        emb            = OpenAIEmbeddings(model="text-embedding-3-small")
        splitter       = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=64)
        chunks         = splitter.split_documents(docs)

        # Hybrid retrieval
        vectorstore    = Chroma.from_documents(chunks, emb)
        bm25           = BM25Retriever.from_documents(chunks, k=10)
        dense          = vectorstore.as_retriever(search_kwargs={"k": 10})
        self.retriever = EnsembleRetriever(retrievers=[bm25, dense], weights=[0.3, 0.7])
        self.reranker  = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

        PROMPT = ChatPromptTemplate.from_messages([
            ("system", """Answer based ONLY on the context. Cite sources as [Source N].
If the answer is not in the context, say: "I don't have information about this."

Context:
{context}"""),
            ("human", "{question}"),
        ])
        self.chain = (
            {
                "context":  lambda x: self._retrieve_and_rerank(x["question"]),
                "question": RunnablePassthrough(),
            }
            | PROMPT
            | self.llm
            | StrOutputParser()
        )

    def _retrieve_and_rerank(self, query: str, top_k: int = 5) -> str:
        candidates = self.retriever.invoke(query)
        pairs      = [(query, d.page_content) for d in candidates]
        scores     = self.reranker.predict(pairs)
        ranked     = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)[:top_k]
        return "\n\n".join(
            f"[Source {i+1}] {doc.page_content}"
            for i, (doc, _) in enumerate(ranked)
        )

    def query(self, question: str) -> dict:
        t0     = time.perf_counter()
        answer = self.chain.invoke({"question": question})
        return {"answer": answer, "latency_ms": round((time.perf_counter() - t0) * 1000)}

The best RAG systems are built iteratively: start with a simple baseline (fixed-length chunks, single-vector retrieval, no reranker), measure with RAGAS, identify the weakest layer, and improve that layer first. For challenging queries, Corrective RAG, Self-RAG, and Agentic RAG patterns provide increasingly powerful retrieval strategies. Avoid adding complexity before you have evidence that a simpler approach has failed — and always maintain evaluation infrastructure so that each improvement can be verified quantitatively. The optimization roadmap and production code template above provide a practical path from baseline to production-grade RAG that applies to most enterprise knowledge retrieval use cases.

RAG System Engineering Principles

  1. Evaluate each layer independently: Retrieval quality, reranker quality, and generation quality are distinct metrics
  2. Never skip the reranker for production: Vector similarity is fast but imprecise; reranking is cheap and accurate
  3. Hybrid search is the baseline: BM25 handles exact matches that dense retrieval misses
  4. Chunk size affects everything: Too small = semantic fragmentation; too large = irrelevant content injected
  5. Faithfulness is your primary SLO: A fast, relevant, but hallucinated answer is worse than no answer
  6. Metadata is first-class: Filter by date, source, department, access level — don’t let retrieval be purely semantic
  7. Context ordering matters: Critical evidence belongs at the beginning or end, not the middle
  8. Corrective RAG is your insurance: When retrieval fails, web search or abstention is better than hallucination

A production RAG system that implements all eight principles will consistently outperform one that relies on a better model alone.

RAG Deep Dive Conclusion

RAG is the most widely deployed pattern for connecting LLMs to organizational knowledge. At its core, it is a retrieval engineering problem as much as a generation engineering problem. The document chunking strategy, embedding model, indexing configuration, hybrid search fusion, reranker, and prompt construction each contribute measurably to the final quality. The RAGAS evaluation framework makes these contributions measurable and comparable across experiments. The advanced patterns—Corrective RAG, Self-RAG, Agentic RAG, and Graph RAG—provide escalating levels of sophistication for queries that straightforward vector retrieval cannot answer. Build iteratively, measure everything, and never mistake retrieval quality for generation quality.

RAG Deep Dive: Key Takeaways

  • Hybrid search (BM25 + dense) consistently outperforms pure vector search
  • Always add a cross-encoder reranker before injecting context—it’s cheap and effective
  • Chunk size affects everything: start with 512 tokens, 64 overlap, and tune from there
  • Faithfulness is your primary quality SLO; hallucinations hurt more than gaps
  • Evaluate retrieval independently from generation—they fail for different reasons
  • Corrective RAG (web search fallback) improves recall on out-of-knowledge-base queries
  • For enterprise knowledge bases, Graph RAG handles relationship-heavy queries that vector search cannot
  • RAGAS provides the standard evaluation suite: faithfulness, relevance, precision, recall

RAG System Quick Reference

Chunking Strategy Selection

Document TypeStrategychunk_sizechunk_overlap
Legal contractsStructure-aware (by section)1000100
Technical manualsStructure-aware (by heading)51264
News articlesFixed-length51264
Research papersHierarchical (abstract + sections)80080
Code documentationFile-level + function-level25632
Product FAQsQ&A aware (split on questions)2560
Mixed corpusSemantic chunking400–60050

RAGAS Score Targets

MetricAcceptableGoodExcellent
Faithfulness> 0.70> 0.80> 0.90
Answer Relevancy> 0.65> 0.75> 0.85
Context Precision> 0.60> 0.70> 0.80
Context Recall> 0.55> 0.70> 0.80

Production Engineering Notes

Building production LLM systems requires continuous learning and adaptation. The patterns and tools in this ecosystem evolve rapidly, but certain engineering principles remain constant:

On reliability: The most reliable systems are built with defense in depth � multiple independent validation layers, graceful degradation paths, circuit breakers, and comprehensive observability. No single component should be a single point of failure.

On evaluation: Automated evaluation enables speed; human evaluation provides ground truth. Calibrate your automated judges regularly against human labels. A 10-point quality regression detected in CI costs hours to fix; the same regression discovered after deployment costs days or weeks of user trust.

On iteration: LLM systems improve through measurement, not intuition. Every prompt change, model upgrade, and retrieval configuration should be evaluated against a representative dataset before deployment. Blind experimentation is expensive; instrumented experimentation is invaluable.

On operations: Monitor cost, latency, and quality as a unified picture. A system that is fast and cheap but inaccurate is not production-grade. A system that is accurate but 10x over budget is not sustainable. The engineering goal is acceptable quality at acceptable cost within acceptable latency.

RAG is the most adopted LLM design pattern in production. The RAGAS leaderboard and MTEB benchmark are the standard references for tracking retrieval quality improvements over time.

RAG Resources

LLM
LLM RAG Retrieval Chunking Embeddings Hybrid-Search Reranking Vector-DB
This post is licensed under CC BY 4.0 by the author.