A runbook assistant can retrieve the wrong procedure perfectly. If an old document describes a retired service, better similarity search only makes the wrong answer easier to find.
Retrieval-augmented generation combines a language model with retrieved information. The original RAG research established an important architecture for knowledge-intensive tasks, but retrieval does not guarantee that the material is current, authorized, or appropriate for this environment.
Fix the document contract first
I would require an owner, service scope, environment, last-reviewed date, and a way to identify superseded content. A document titled “restart procedure” is not enough. The assistant needs to know which service and deployment model it applies to.
Use an illustrative conflict: one guide says to restart a process on a host; the newer guide describes a managed runtime where that action is unavailable. Returning both paragraphs without explaining the conflict is not useful evidence.
Retrieval must respect the caller
Apply access controls before restricted material enters the model’s context. Filtering a generated answer after the model has already received a sensitive document is not the same boundary. Keep permissions aligned as documents move, owners change, and indexes are rebuilt.
The same care applies to logs attached to a request. A convenient troubleshooting bundle can contain credentials or personal data. Decide what the assistant is permitted to receive rather than assuming everything readable by the collection service is appropriate for every user.
Ask for citations that can be checked
A link is useful only when it supports the claim beside it. For a procedural answer, cite the document section and version that contains the relevant step. If the retrieved evidence conflicts or lacks environment context, the assistant should say so and ask for clarification.
Treat retrieved prose as data, not instructions that can change tool permissions. OWASP notes that retrieval does not eliminate prompt-injection risk; an indexed document can still contain hostile directions.
Evaluate retrieval separately
Test whether the correct current document appears, whether obsolete content is excluded, and whether unauthorized documents stay out. Then evaluate the generated answer. Otherwise, a generation score hides the reason the assistant failed.
The useful outcome is a shorter path to the right runbook. Sounding certain about a stale procedure is movement in the opposite direction.