Production RAG Architecture: Chunking, Hybrid Retrieval, Reranking, and Evaluation

Diagnose before you tune

When answers are wrong, there are only three places to look, and they need different fixes. Determine which one you have before changing anything, because teams routinely spend weeks on prompt engineering to fix a retrieval bug.

1. The right chunk was never retrieved. This is a retrieval problem: fix chunking, embeddings, or search strategy. It is the majority of failures.
2. The right chunk was retrieved but ranked below the context cutoff. This is a ranking problem: add a reranker.
3. The right chunk was in context and the model still answered wrong. Only now is this a prompting or model-capability problem.

Instrument this directly. Log, for every query, the retrieved chunk IDs and their scores, and for your eval set annotate which chunk contains the answer. Recall@k against that annotation is the number that tells you which of the three problems you actually have.

Chunking is a document-structure problem

Fixed-size chunking with a character count is the default in every tutorial and it is wrong for almost every real corpus. It splits tables in half, severs headings from the text they govern, and leaves clauses referring to definitions that landed in another chunk.

• Split on structure first — headings, sections, list boundaries, table units — and only fall back to size limits within a structural unit.
• Prepend breadcrumb context to every chunk: document title, section path, and effective date. A chunk that reads 'the rate shall be 4.5%' is useless; 'Loan Agreement > Section 3 > Interest: the rate shall be 4.5%' is retrievable.
• Keep tables whole and render them as markdown, with the header row repeated if you must split.
• Overlap modestly — 10 to 15% — and only across prose, never across table rows.
• Store the parent document and offsets alongside each chunk so you can expand context at generation time without re-retrieving.

A pattern worth the extra complexity: retrieve on small chunks, generate on large ones. Small chunks embed more precisely; large parent sections give the model enough context to reason. Index the small, fetch the parent, pass the parent.

Hybrid retrieval, then rerank

Dense embeddings capture meaning and miss exact tokens — part numbers, error codes, ticker symbols, statute references, proper nouns your model never saw. Lexical search (BM25) is the opposite. Running both and fusing the results is the single highest-leverage change in most RAG systems, and it is a day of work.

// Reciprocal rank fusion: no score normalization needed across
// two retrievers whose scores are not comparable.
function fuse(dense: string[], lexical: string[], k = 60) {
  const scores = new Map<string, number>();
  for (const list of [dense, lexical]) {
    list.forEach((id, rank) => {
      scores.set(id, (scores.get(id) ?? 0) + 1 / (k + rank + 1));
    });
  }
  return [...scores.entries()]
    .sort((a, b) => b[1] - a[1])
    .map(([id]) => id);
}

Then rerank. Retrieve wide — 50 to 100 candidates — and pass them through a cross-encoder reranker that scores each chunk against the query jointly rather than comparing precomputed vectors. Keep the top 5 to 10. This costs latency measured in tens of milliseconds and typically buys more accuracy than upgrading your embedding model.

Grounding: make citation structurally required

Do not ask the model politely to cite sources. Make an uncited answer impossible to represent. Give each chunk an ID in the context, require the response as structured output where every claim carries a source ID, and validate server-side that each ID exists in the retrieved set before showing the answer to a user.

const response = await client.messages.create({
  model: "claude-sonnet-5",
  max_tokens: 2048,
  system: [
    { type: "text", text: SYSTEM_PROMPT, cache_control: { type: "ephemeral" } },
  ],
  messages: [{ role: "user", content: buildContext(chunks, question) }],
  tools: [answerSchema],       // forces {claims: [{text, sourceIds}]}
  tool_choice: { type: "tool", name: "answer" },
});

// Reject silently-hallucinated citations before they reach the user.
const valid = new Set(chunks.map((c) => c.id));
const bad = answer.claims.flatMap((c) => c.sourceIds).filter((id) => !valid.has(id));
if (bad.length) throw new UngroundedAnswerError(bad);

Equally important: give the model an explicit exit. 'If the context does not contain the answer, say so and stop' converts a hallucination into a miss, and a miss is a retrieval bug you can measure and fix. Systems without that instruction have no floor.

Cost and latency controls

• Prompt caching on the system prompt and any stable context block — for repeated document QA this is often a 5–10x cost reduction on input tokens and a meaningful latency win.
• Semantic caching on the query side: normalize and embed the question, and serve a stored answer on a near-exact hit. Support corpora have enormous duplicate query rates.
• Route by difficulty. A small fast model handles the majority of lookups; escalate to a larger model when the reranker's top score is low or the question requires synthesis across many chunks.
• Stream tokens to the UI, and stream retrieval status before them. Perceived latency is dominated by time-to-first-token, not total time.
• Batch and pre-embed at ingest, never at query time.

The eval loop is the product

Without evaluation you are tuning by vibes, and vibes reliably regress. Build a set of 100 to 300 real questions with known correct sources before you optimize anything — real user questions, not ones you invented, because invented questions use your vocabulary and real ones do not.

• Retrieval metrics: recall@k and MRR against annotated gold chunks. These are cheap, deterministic, and catch most regressions.
• Groundedness: does every claim trace to a retrieved chunk? Automatable with an LLM judge and reliable enough to gate deploys.
• Answer quality: LLM-as-judge with a rubric, sampled and spot-checked by a human weekly. Never fully trusted, but directionally sound.
• Run the whole suite in CI on every prompt, model, or chunking change. The point is not the absolute score — it is knowing that today's change made yesterday's answers worse.