Miguel Rodrigues logoMiguel Rodrigues

| 1 min read

What I Learned Building a RAG System

Practical notes from shipping a retrieval-augmented generation workflow that needed stable latency, predictable grounding, and honest failure behavior.

  • AI
  • RAG
  • Backend

I used to think the hard part of RAG was model quality. In production, the harder part was control.

The model is only one stage in a longer pipeline. Every stage can fail in a different way: indexing, retrieval, ranking, context assembly, and generation. If you do not instrument each stage independently, debugging turns into guesswork.

Grounding Is a Product Feature

The best prompt in the world does not help if retrieval is weak. I ended up spending more time improving chunking and metadata than adjusting prompts.

Key wins:

  • Smaller chunks with overlap improved semantic recall.
  • Versioned documents avoided stale context when content changed.
  • Re-ranking reduced irrelevant but semantically "close" matches.
  • Strict source attribution improved user trust.

Latency Budgeting

The system felt slow when I treated every request as equal. Latency became manageable after introducing an explicit budget per stage.

const budget = {
  retrieveMs: 120,
  rerankMs: 80,
  generateMs: 900,
};

const result = await runRagPipeline({
  query,
  budget,
  maxContextTokens: 1800,
});

If retrieval exceeded budget, I returned fewer chunks instead of failing silently. Imperfect but fast was usually better than "eventually correct."

Failure Modes Worth Handling

Three failure modes mattered most:

  1. No good retrieval candidates.
  2. Good retrieval but weak context assembly.
  3. Hallucinated confidence when evidence was thin.

For each one, I added explicit response strategies:

  • Return "insufficient context" instead of over-answering.
  • Ask clarifying questions when intent was broad.
  • Show cited sources in the UI every time.

Final Takeaway

Reliable RAG is mostly systems design. The product becomes better when you treat retrieval quality, observability, and fallback behavior as first-class concerns, not as glue code around a model call.