To provide a structured method for testing Retrieval-Augmented Generation (RAG) agents across accuracy, latency, sourcing, formatting, and robustness. This SOP ensures consistency, identifies weaknesses, and provides actionable feedback to developers.
Step 1: Access & Setup
Before testing, confirm access to:
- RAG Agent Interface (chat UI or API).
- QA Record Sheet ( link to the file).
- RAG Knowledge Base (files being tested).
- LLM Model/ChatGPT (for generating questions and cross-checking answers).
- Reviewed and understood SOP Video (Optional): Loom Walkthrough.
https://www.loom.com/share/469ca00e76d84b75830d7cc8f937061f?sid=bf455a42-3960-4ce8-8964-ac9a9256489a
Before testing Rag Agent, have a QA Testers checklist ready and save it under the client's folder and name it as Rag QA, and name each tab, QA Date ( mm/dd/yy) by user.
RAG QA Checklist.xlsx
For example, QA 09/22/25 by Zulfiya F.
Evaluate each QA item as Pass, Fail or Not Tested.
Step 2: How to QA RAG and what kind of questions to ask
Manual Questions: Read files directly and create factual, contextual, and analytical questions.
- AI-Assisted Questions: Use ChatGPT or another LLM to generate a broad question set.
- Best Practice: Use both methods—manual ensures accuracy, AI ensures coverage.
- Save each question in the QA sheet.
Step 3: Test prompt execution and record findings
For each question asked of the RAG agent: