RAG in production with Azure AI Search and embeddings
Chunking, reranking and latency: what changes from notebook to cluster.
RAG in a notebook is retrieval + prompt. In production on Azure AI Search, latency, cost, and quality depend on infrastructure choices — not just the model.
Chunking and index
Chunks that are too small lose context; too large dilute the embedding. Index metadata (slug, section, language) to filter before rerank.
Reranking and latency
A second rerank stage improves precision but adds tens of ms. Measure end-to-end p95: query embedding + search + rerank + generation. Cache frequent queries in Redis or in the index itself.
Observability
Log queryId, retrieved chunks, and score — without storing PII. Compare answers to offline ground truth before changing chunk size in production.
Written by Luis Soares — software architect, on the road to Microsoft MVP.