Retrieval-augmented generation succeeds or fails before the final prompt. If the system cannot locate the right evidence, preserve access controls, distinguish current from obsolete material, and expose provenance, a fluent answer only hides the defect.
Treat the retrieval layer as a governed data product with consumers, contracts, quality measures, freshness expectations, and an owner. The model is downstream of that product.
Define the retrieval unit around the decision
Fixed-size chunks are a baseline, not a universal design. Contracts, manuals, tickets, source code, and tabular records have different semantic boundaries. Preserve headings, document identity, version, access policy, and surrounding context so retrieved evidence can be interpreted.
Create a query set from real user tasks and known difficult questions. Label which evidence is necessary, merely useful, or misleading. This dataset enables retrieval evaluation without conflating it with generation.
Build a permission-aware ingestion path
Every indexed object needs lineage back to its source and access policy. Deletion, retention, and permission changes must propagate. Otherwise the RAG system becomes a shadow repository that returns information users can no longer access in the source system.
- Source identity, version, and ingestion timestamp
- Document and section boundaries
- Principal or group access rules
- Retention and deletion behavior
- Embedding and parser versions
- Quality checks for empty, duplicated, or malformed content
Measure retrieval before answer quality
Track whether required evidence appears in the candidate set and at what rank. Segment results by source, document type, query class, recency, and permission condition. Only then evaluate grounded answer quality, citation correctness, omission, and refusal when evidence is insufficient.
Hybrid retrieval, reranking, metadata filters, and query decomposition can improve particular failure modes. Add them because error analysis identifies a need, not because they appear on an architecture diagram.
Operate freshness and regressions
Content changes continuously. Monitor ingestion lag, failed parses, permission drift, index coverage, and retrieval regressions. Version the evaluation set and replay it whenever chunking, embeddings, reranking, or source connectors change.
Treat retrieval quality as an owned product
A production retrieval layer needs explicit ownership of source onboarding, parsing, permissions, chunking, indexing, freshness, and deletion. The team should know which source is authoritative, how quickly a change becomes searchable, and how access revocation propagates. Without that operating model, retrieval quality degrades quietly while the application continues to produce fluent answers.
Evaluate the stages separately. Corpus coverage asks whether the needed evidence exists. Retrieval evaluation asks whether the system found it. Context assembly asks whether useful evidence survived ranking and formatting. Answer evaluation asks whether the model used it faithfully. A single end-to-end score cannot tell the team which stage to repair.
Engineer permissions and provenance into the index
Permission filtering must occur before evidence reaches the model, not after an answer is generated. Carry source identity, tenant, access labels, effective dates, and deletion state through parsing and indexing. Test with users whose entitlements differ by only one sensitive document; broad role tests often miss the boundary failures that matter.
Every answer should retain machine-readable provenance: retrieved identifiers, versions, ranking scores, filters, and the exact context provided to the model. Citations in the interface are useful, but they are not a substitute for an audit record. Provenance lets the team reproduce a failure after documents or indexes have changed.
Executive decision record
The decision is whether retrieval can supply authorized, current, and sufficient evidence for the target questions within the product latency budget. Write that decision before selecting a model, vendor, framework, or implementation pattern. A written boundary keeps technical exploration connected to the operating outcome and makes it possible to explain why the organization advanced, revised, or stopped the work.
Approval should depend on judged queries, corpus coverage, permission tests, freshness and deletion checks, stage-level retrieval measures, and answer-grounding evaluation. The evidence does not need to remove every uncertainty, but it should address the uncertainty capable of changing value, architecture, risk, or ownership. Record the baseline, assumptions, unresolved questions, and the person accepting the next stage.
Failure boundary and operating ownership
The central failure to guard against is treating a vector search demo as a maintained data product while source authority, provenance, and access changes drift silently. Treat that condition as a testable scenario. Define how the system detects it, what users experience, which action is prevented or reversed, and what evidence reaches the person responsible for recovery.
Long-term accountability sits with a named retrieval or knowledge-product owner working with source-system owners and the application team. Supporting specialists can provide platforms, research, review, or delivery capacity, but they cannot substitute for an owner who controls policy and operating change. Name that owner before production and include the ownership path in release evidence and incident procedure.
A practical 90-day application plan
During the first 30 days, convert judged queries, corpus coverage, permission tests, freshness and deletion checks, stage-level retrieval measures, and answer-grounding evaluation into a bounded evidence plan. Assign each artifact to a named contributor, identify the representative inputs required, and agree on the comparison baseline before implementation expands. The objective of this period is to expose the assumption most likely to invalidate the work while the cost of changing direction is still low.
During days 31 through 60, build or instrument the smallest complete workflow that can support the decision about whether retrieval can supply authorized, current, and sufficient evidence for the target questions within the product latency budget. Include the real data and authorization path where feasible, record exceptions, and review difficult cases with the people who own the underlying process. Resist adding breadth until the team can explain the measured behavior of this narrow slice.
During days 61 through 90, test the boundary represented by treating a vector search demo as a maintained data product while source authority, provenance, and access changes drift silently. Exercise degraded dependencies, ambiguous inputs, recovery, and handoff rather than demonstrating only successful cases. End the period with a written advance, revise, or stop decision that cites evidence, residual exposure, expected operating cost, and the next authority boundary.
The review should be accepted by a named retrieval or knowledge-product owner working with source-system owners and the application team. That group should confirm not only that the system can work, but that ownership, support capacity, monitoring, and change control are credible. If those conditions are absent, the responsible outcome is another bounded learning stage rather than an unsupported production commitment.
Practical checklist
- Identify the evidence required for representative questions.
- Preserve source, version, and authorization metadata.
- Evaluate retrieval independently from generation.
- Require citations that resolve to accessible source passages.
- Test deletion and permission changes end to end.
- Monitor freshness, coverage, and failed ingestion.
- Define refusal behavior when evidence is missing.
Engagement scenario
A support organization indexes product documentation, resolved tickets, and current policy. Retrieval evaluation reveals that obsolete tickets outrank current policy for one product family. The team corrects source weighting and version filters before tuning answer style, preventing a polished interface from shipping incorrect guidance.
Continue reading
- [RAG versus long context](/rag-vs-long-context-enterprise-ai)
- [Data engineering for production AI](/data-engineering-for-production-ai)
- [LLM evaluation framework](/llm-evaluation-framework-production)
Sources and further reading
- [Lost in the Middle research](https://arxiv.org/abs/2307.03172)
- [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework)