Evaluating RAG with RAGAS

Alexia Schneider
alexia.schneider@umontreal.ca

2025-11-03

RAG

Key point: Retrieval Augmentated Generation is an architecture for AI-integrated systems. It relies on an external knowledge base to improve on the overall quality of a generative LLM’s output.

a foundational architecture in modern LLM systems” —(Shankar and Husain 2025)

Description of the RAG architecture from Weaviate (2025)

Basic schema of a RAG system, an architecture or pipeline for LLM-integrated systems

What limitations of foundational LLMs can RAG solve ?

What limitations of foundational LLMs can RAG solve ?

  • Reliability and transparency (for the dev): providing a specialized and human verified external knowledge source ensures that the model will output based on those elements rather than it’s implicit knowledge base.
  • Better reproducibility: traceable sources and choice of IR strategy.
  • Preservation of the general-purpose and adaptability of foundational LLMs: doesn’t require special training or fine-tuning on a specific task.

What can be some limits of RAG ?

What can be some limits of RAG ?

Higher level problems:

  • Dataset biais: External knowledge base = dataset with its bias and limitations too.
  • Cost: Adding to regular prompt engineering: encoding the external knowledge base, DBB storage, eval and maintenance of a complex system, longer prompts containing extra info.
  • Latency: pipeline = increased response time.
  • Complexity: Many parameters = complex evaluation and optimization: IR strategy (TF-iDF/BM25 or semantic search with vector proximity, neural retrievers), chunk size, number of chunks provided and ranking algorithm (top-k).

Internal problems:

  • Information dissemination: what if the chunks needed end up being too big or numerous for the LLM to be competent?
  • ‘Blind spot’ effect: if the key information is in a chunk not retrieved, then that part is lost to the user.
  • Evaluation : each step of the pipeline can cause failures

Evaluating RAG: when and where

Checkpoints for evaluation: A. Chunk size and type. B. Retrieval strategy. C. Reranker algorithm. D. Final output considering the entire flow.

Shankar and Husain (2025) recommendations for RAG Evaluation

General recommendations for evaluation:

  • Identify where the pipeline weakens (or where it breaks)
  • Evaluate at each step
  • Form an evaluation dataset of expected queries, retrieved context and gold-standard answers

Reference-based evaluation pros and cons

pros cons
control of output from expected user behaviour user query might not fit expected queries ( e.g. edge cases and jailbreak attempts)
fixed dataset means that different prompting strategies can be assessed over time annotation is time consuming

What are some tasks where a reference answer or ‘gold-standard answer’ might not be available ?

What are some tasks where a reference answer might not be available ?

  • Open-ended queries
  • Summarization
  • Interpretative tasks
  • Changing context

Proposed solution: RAGAS

Article

Submitted on 26 Sep 2023 (v1), last revised 28 Apr 2025 (this version, v2) on arXiv.

Authors

Es et al. (2025)
  • Exploding Gradients : private company that built RAGAS (open-source oriented)
  • CardiffNLP : Schockaert: worked on EvoPrompt.
  • AMPLYFI: private company for market insights

RAGAS in a nutshell

Key element: Reference-free (not tied to having ground truth available) evaluation framework for retrieval agumented generation

Essentially: a series of prompts to decompose the evaluation for specific aspects of the RAG (faithfulness, answer relevance, context relevance).

Integration: llama-index and Langchain.

Available: https://github.com/explodinggradients/ragas

Ratings: 11.1k stars on GitHub

RAGAS in a nutshell

RAGAS suggests an evaluation at the final step (D)

Evaluation of the final output

  • Faithfulness: The answer should be grounded in provided context.
    • hallucinations
    • omissions
    • misinterpretation

Answer Relevance: The generated answer should address the actual question.

Context Relevance: Retrieved context should be focused, containing as little irrelevant information as possible.

Faithfulness eval with RAGAS

The answer should be grounded in provided context

Example: You have designed a RAG system to help you with your courses notes. You want to make sure that the IA-system answers based on the content of the course as it is very advanced knowledge and not from unreliable sources (the web).

Strategy:

  1. Create statements from the output given a user query.

Prompt:

Given a question and answer, create one or more statements from each sentence in the given answer. \n question: [question] answer: [answer]

Question = query = “Can you summarize last week’s course?”

Answer = output =

“Here’s a concise summary of your Week 8 – AI Agent Architecture course content:

  • Agents are autonomous systems that:

    • Interact not only with users but also with each other and their environment.
    • Go beyond simple LLM + tool use (sometimes called a “single agent”).”

Statements created for evaluation =

  1. “The course discussed in Week 8 is titled”AI Agent Architecture.”

  2. “The week focused on the structure, interaction, and evolution of AI agents.”

  3. “Agents are autonomous systems.”

  4. “Agents interact with users, other agents, and their environment.”

  1. LLM-as-judge to evaluate the statement against the retrieved context.

Prompt:

Consider the given context and following statements, then determine whether they are supported by the information present in the context. Provide a brief explanation for each statement before arriving at the verdict (Yes/No). Provide a final verdict for each statement in order at the end in the given format. Do not deviate from the specified format. statement: [statement 1] ... statement: [statement n]

Context: course.md week1, week8, week n, syllabus.xslx chunk 1, chunk n.

Statement to assess : “The course discussed in Week 8 is titled”AI Agent Architecture.”

Evaluation output : “Chunk 1 of the Syllabus provided does mention that the topic of Week 8 is AI Agent Architecture. Verdict : Yes”

Metric : supported statements/total number of statements

Answer Relevance

The generated answer should address the actual question.

Strategy: Retro-engineering what the question could have been from the answer.

Generate a question for the given answer. answer: [answer]

From the output summary:

Possible Question: “What are the key concepts, models, and research papers covered in Week 8 – AI Agent Architecture, and how do they illustrate the evolution from single LLM-based systems to multi-agent ecosystems?”

To be compared with the actual query: “Can you summarize last week’s course?”

Metric : Cosine similarity between the actual question and the generated answer.

Context Relevance

Retrieved context should be focused, containing as little irrelevant information as possible.

Strategy: Extracting sentences from the context that have could have been used to give the answer.

Prompt:

Please extract relevant sentences from the provided context that can potentially help answer the following question. If no relevant sentences are found, or if you believe the question cannot be answered from the given context, return the phrase "Insufficient Information". While extracting candidate sentences you’re not allowed to make any changes to sentences from given context.

Helps analyse the granularity of the chunks provided and noise in the top-k (tells you whether there is anything relevant or not in the context).

Metric : number of extracted sentences/total number of sentences

Evaluation of the evaluation method

Evaluation Dataset

WikiEval Dataset : created for the purpose of RAGAS.

Question/Answers created from 50 wiki pages (after 2022 to avoid reliance on implicit knowledge).

  1. Creation of the Question

Prompt:

Your task is to formulate a question from given context satisfying the rules given below: 1. The question should be fully answered from the given context. 2. The question should be framed from a part that contains non-trivial information. 3. The answer should not contain any links. 4. The question should be of moderate difficulty. 5. The question must be reasonable and must be understood and responded to by humans. 6. Do not use phrases that ’provided context’, etc in the question context:

  1. Generation of the Answer

Prompt:

Answer the question using the information from the given context. question: [question] context: [context]

  1. Annotation by 2 humans with interannotation agreement reaching 95% (faithfulness and context relevance) and 90% (answer relevance).

Evaluation of the evaluation method

Their evaluation relies on the agreement of the method with a human annotator based on pairwise comparisons != absolute scores.

question answer A answer B context
When is the scheduled launch date and time for the PSLV-C56 mission, and where will it be launched from? The PSLV-C56 mission is scheduled to be launched on Sunday, 30 July 2023 at 06:30 IST / 01:00 UTC. It will be launched from the Satish Dhawan Space Centre, Sriharikota, Andhra Pradesh, India. The scheduled launch date and time for the PSLV-C56 mission have not been provided.The PSLV-C56 mission is an important space mission for India. It aims to launch a satellite into orbit to study weather patterns.’ “The PSLV-C56 is the 58th mission of Indian Space Research Organisation’s Polar Satellite Launch Vehicle (PSLV) and the 17th flight of the PSLV-CA variant, and will be get launched from Satish Dhawan Space Centre First Launch Pad ( FLP ).is Scheduled to get launched on Sunday, 30 July 2023 at 06:30 IST / 01:00 UTC from Satish Dhawan Space Centre, Sriharikota, Andhra Pradesh, India. This is a dedicated commercial mission through NSIL with DS-SAR as primary satellite and VELOX-AM as a co-passenger satellite With other 5 Satellites, All satellites from this mission belongs to Singapore.”

Ragas is compared to :

  • GPT Score : for faithfulnes, range 0-10.

GPT Score Prompt:

Faithfulness measures the information consistency of the answer against the given context. Any claims that are made in the answer that cannot be deduced from context should be penalized. Given an answer and context, assign a score for faithfulness in the range 0-10. context: [context] answer: [answer]

  • GPT Ranking: for answer relevancy, ranking.

GPT Ranking Prompt:

Answer Relevancy measures the degree to which a response directly addresses and is appropriate for a given question. It penalizes the present of redundant information or incomplete answers given a question. Given an question and answer, rank each answer based on Answer Relevancy. question: [question] answer 1: [answer 1] answer 2: [answer 2]

Evaluation results

Evaluation of 3 evaluation methods on the WikiEval dataset

« The results in Table 1 show that our proposed metrics are much closer aligned with the human judgements than the predictions from the two baselines.

For faithfulness, the Ragas prediction are in general highly accurate.

For answer relevance, the agreement is lower, but this is largely due to the fact that the differences between the two candidate answers are often very subtle.

We found context relevance to be the hardest quality dimension to evaluate. In particular, we observed that ChatGPT often struggles with the task of selecting the sentences from the context that are crucial, especially for longer contexts. » (Es et al., 2025, p. 5) (pdf)

Other methods of RAG evaluation

Reference-based score:

  • BERTScore
  • MoverScore

-> Comparison between vector of output and “golden standard” answer.

Reference-less score:

  • Asking ChatGPT to score the output (scale of 0 to 100) -> GPT Score and GPT Ranking
  • ARES

Difference between Automated RAG Evaluation System ARES framework and RAGAS

Saad-Falcon et al. (2024) (a year after RAGAS).

Synthetic data generation (generating query‑passage‑answer triples) + fine‑tuned “judge” models + a small human annotated set + Prediction‑Powered Inference (PPI) to give confidence intervals, better ranking of RAG systems.

Comment: much more complex to set-up, but better for robust eval (confidence intervals)

Would you use RAGAS to evaluate your RAG?

Personal comments on RAGAS

Strengths:

  • Reproducibility: they provide the prompts in full
  • Clever workaround for the context relevance (with cosine similarity between query and retroengineered query).
  • Easy and intuitive implementation.

Limits:

Evaluation of the evaluation method:

  • The model used to create the dataset and to perform the ragas seems to be the same (namely ChatGPT without stating its version) : bias towards better performances at it would agree with itself more and create answer and questions that resemble more the kind of ‘statements’ it would generate.
  • Compared evaluation methods, GPT Score and GPT Ranking are not strong LLM-as-judge methods.
  • Limited dataset of 50 Q&A.

RAGAS as evaluation: - Splitting the answer into statements (faithfulness score) adds another layer of interpretation by the LLM, increasing chances of hallucinations etc. - Lack of certainty assessment.

References

Es, Shahul, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2025. “Ragas: Automated Evaluation of Retrieval Augmented Generation.” arXiv. https://doi.org/10.48550/arXiv.2309.15217.
Lewis, Patrick, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, et al. 2021. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” arXiv. https://doi.org/10.48550/arXiv.2005.11401.
Saad-Falcon, Jon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2024. “ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems.” In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), edited by Kevin Duh, Helena Gomez, and Steven Bethard, 338–54. Mexico City, Mexico: Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.naacl-long.20.
Shankar, Shreya, and Hamel Husain. 2025. Application-Centric AI Evals for Engineers and Technical Product Managers.
Weaviate. 2025. “Advanced RAG Techniques.” Weaviate Ebook.