Skip to content

RAG Evaluation & Testing — The Definitive Metrics Whitepaper

What this is: an exhaustive, production-grade reference for evaluating Retrieval-Augmented Generation systems — the mathematics, execution mechanics, edge cases, and architectural impact of every metric that matters. Written for hands-on implementation and advanced AI/QA interviews.

Companion guides: RAG Automation Testing Roadmap (the CI process), AI Test Strategy §2 (strategy view), Ragas FAQ (the framework). This page is the metric-level deep dive those pages reference.

1. Executive Summary — The Structural Dichotomy of RAG Failures

A RAG system is not one model; it is a pipeline with two independently-failing halves. Every production incident traces to one of two surfaces, and conflating them is the single most expensive mistake in RAG QA.

                       ┌──────────────────────── RAG PIPELINE ────────────────────────┐
   query ──▶ [embed] ──▶ [vector search] ──▶ [rerank] ──▶ [context] ──▶ [LLM] ──▶ answer
             └──────────── RETRIEVAL (the "R") ───────────┘   └──── GENERATION (the "G") ────┘
                     failure surface #1                              failure surface #2

The R surface owns whether the correct evidence reaches the context window. Its failures are silent — the pipeline returns a fluent answer regardless of whether the right chunk was fetched. R-failures are measured with Information Retrieval metrics (Precision@K, Recall@K, MRR, NDCG) that require a labelled ground-truth set of relevant chunks.

The G surface owns whether the LLM faithfully uses the evidence it was given. Its failures occur even when retrieval is perfect: the generator can ignore the context window, blend it with parametric memory (facts baked into weights during pre-training), or over-summarize away the critical qualifier. G-failures are measured with inference-layer metrics (Faithfulness, Answer Relevance) that a judge LLM computes against the actually-retrieved context.

The cascade problem — why a great retriever still produces wrong answers

The two surfaces are multiplicative, not additive. If retrieval succeeds 90% of the time and generation faithfully uses good context 90% of the time, end-to-end correctness is ≈ 0.9 × 0.9 = 0.81, not 0.9. Worse, the failure modes interact:

Scenario Retrieval Generation End-user sees Root cause you must isolate
A ✅ correct chunk fetched ✅ grounded Correct answer —
B ✅ correct chunk fetched ❌ ignored context, used parametric memory Plausible but wrong/outdated answer G-failure (hallucination from weights)
C ❌ chunk missing from top-K ✅ faithful to what it got Confident answer built on wrong evidence R-failure (context precision)
D ❌ nothing relevant retrieved ✅ faithful → refuses/says "I don't know" Honest non-answer R-failure (context recall) — correct generator behaviour

The load-bearing insight: Scenario B and Scenario C produce identical symptoms to a black-box tester — a wrong answer — but require opposite fixes (prompt/generator hardening vs. retrieval/chunking tuning). This is why RAG evaluation must decompose the pipeline. A single end-to-end "accuracy" number tells you that you failed, never where. Every metric below exists to localize failure to one surface.

Parametric vs. non-parametric memory is the crux of G-failures. The LLM holds two knowledge stores: the retrieved context (non-parametric, in the prompt) and its training weights (parametric). A faithful RAG generator must suppress parametric recall when it contradicts or exceeds the context. Testing for this requires adversarial cases where the ground-truth context deliberately contradicts likely training data (e.g., a fictional company whose "CEO" in the context differs from any real person) — if the model answers from weights, faithfulness collapses.

2. Information Retrieval & Vector-Level Testing (The "R" Surface)

2.1 Vector-Level Testing — Distance Mechanics

Retrieval reduces to a nearest-neighbour search in embedding space. The distance/similarity function is not interchangeable — choosing wrong, or mixing metrics between index-build and query time, silently destroys recall.

        Cosine: angle only            Euclidean (L2): straight-line       Dot Product: angle × magnitude
              q                              q                                    q  (long vector)
             /·                            •                                     /·
            / ·θ  (θ small = similar)     |  \  d = ‖q − c‖₂                    / ·
           /  ·                           |   \                                /  · (favors high-norm c)
          c                              c ────•                              c
    sim = cos θ ∈ [−1, 1]          dist = √Σ(qᵢ−cᵢ)²  ∈ [0, ∞)         dot = ‖q‖‖c‖cos θ ∈ (−∞, ∞)

Mathematical formulations (query vector q, chunk vector c, dimension n):

  • Cosine similarity: cos(q,c) = (Σ qᵢcᵢ) / (‖q‖₂ · ‖c‖₂). Range [−1, 1]. Scale-invariant — measures only orientation. This is the correct default for text embeddings, because sentence length inflates raw magnitude and you want semantic direction, not document size.
  • Euclidean / L2: d(q,c) = √(Σ (qᵢ − cᵢ)²). Range [0, ∞), smaller = closer. Sensitive to magnitude, so a long chunk and short chunk on the same topic land far apart — usually undesirable for text.
  • Dot product / inner product: q·c = Σ qᵢcᵢ. Range (−∞, ∞). Equals cosine only when both vectors are unit-normalized. Rewards high-magnitude vectors, which can be exploited (a chunk with large norm gets retrieved for everything).

The normalization identity that governs everything: for unit-normalized vectors (‖v‖₂ = 1), dot = cosine and L2² = 2 − 2·cosine. This means on normalized vectors all three rank identically — the ONLY differences appear when normalization is inconsistent.

The #1 production bug — normalization mismatch. If embeddings are L2-normalized at index-build time but the query embedding is not normalized (or vice versa), a dot-product index returns garbage rankings while throwing no error. Symptoms: recall craters for no code change after an embedding-library upgrade. Test for it explicitly: assert abs(‖v‖₂ − 1.0) < 1e-6 on a sample of both indexed and query vectors; and run a "self-retrieval" test — embed a chunk, query with that exact text, assert it returns as rank-1 with similarity ≈ 1.0. If self-retrieval fails, normalization or metric config is broken.

Vector-space density & the curse of dimensionality. In high dimensions (768–3072), vectors concentrate: the ratio between nearest and farthest neighbour distances shrinks, so everything looks moderately similar and the top-K becomes noisy. Denser corpora (10M near-duplicate chunks) worsen this — many chunks sit within epsilon of the query. Stress test: measure the similarity gap (score of rank-1 minus score of rank-10) across corpus sizes; a collapsing gap predicts precision degradation before users report it.

Embedding model choice dominates ceiling performance. A 384-dim MiniLM vs. a 3072-dim large model changes both the semantic resolution and the latency/storage cost. Critically, re-embedding the corpus with a new model invalidates every prior similarity threshold — a change that must trigger a full retrieval-regression suite (see RAG Roadmap). Never mix vectors from two embedding models in one index; the spaces are not aligned and cosine between them is meaningless.

2.2 Precision@K and Recall@K

These are the workhorse set-based retrieval metrics. Let K = number of chunks retrieved, R = the full set of ground-truth relevant chunks for a query.

Precision@K = (relevant chunks in top-K) / K          "how much of what I fetched is signal?"
Recall@K    = (relevant chunks in top-K) / |R|         "how much of the signal did I fetch?"

Worked example. Query has |R| = 4 relevant chunks in the corpus. You retrieve K = 5, of which 3 are relevant: - Precision@5 = 3/5 = 0.60 (40% of the context window is noise) - Recall@5 = 3/4 = 0.75 (one critical chunk was missed entirely)

Setting optimal K — the central architectural trade-off. K is not a tuning nicety; it is a direct dial on the precision/recall frontier and on generation quality:

  • K too low → high precision, low recall → context starvation: the answer-bearing chunk sits at rank K+1 and never reaches the LLM. Recall@K is the ceiling on end-to-end correctness — the generator cannot use what it never receives.
  • K too high → high recall, low precision → noise injection: the context window fills with marginally-relevant chunks. This triggers two distinct failures:
  • "Lost in the Middle" — LLM attention is U-shaped: tokens at the start and end of a long context are weighted heavily; the middle is under-attended. A correct chunk buried at position 7 of 15 can be functionally invisible even though it is present. This is an attention/position failure, not a retrieval failure — Recall@K says you succeeded while the answer is wrong.
  • Token cost & latency — every extra chunk is more prompt tokens (linear cost) and more generation latency.

The tuning protocol: sweep K ∈ {3, 5, 8, 10, 15}, plot Recall@K (rising, saturating) against end-to-end Faithfulness (rising then falling as noise/Lost-in-Middle kicks in). The optimal K is the knee — where recall has saturated but generation quality has not yet degraded. This is almost always lower than intuition suggests (frequently K=3–5 with a good reranker), because a reranker lets you retrieve wide (high recall) then trim to a dense, high-precision context.

2.3 Mean Reciprocal Rank (MRR)

Precision@K and Recall@K are order-blind — a relevant chunk at rank 1 scores the same as at rank K. MRR fixes this for the "first correct answer" case.

MRR = (1/|Q|) · Σ (1 / rank_first_relevant_i)     over all queries i in test set Q

For each query, find the rank of the first relevant chunk; the reciprocal is the score (rank 1 → 1.0, rank 2 → 0.5, rank 3 → 0.33, rank 5 → 0.2). Average across queries.

Worked example across 3 queries:

Query 1: first relevant chunk at rank 1  →  RR = 1/1 = 1.000
Query 2: first relevant chunk at rank 4  →  RR = 1/4 = 0.250
Query 3: first relevant chunk at rank 2  →  RR = 1/2 = 0.500
MRR = (1.000 + 0.250 + 0.500) / 3 = 0.583

Why it matters architecturally: MRR is the metric that most directly correlates with reranker quality and with the "Lost in the Middle" defense. Because the LLM attends most strongly to the top of the context, a system that reliably places the answer-bearing chunk at rank 1 (MRR → 1.0) is far more robust than one that buries it at rank 4 (MRR = 0.25) even if both have identical Recall@5. MRR penalizes exactly the failure — a relevant chunk present but low — that set-based metrics are blind to. Edge case: MRR only considers the first relevant hit; for queries needing multiple pieces of evidence synthesized together, MRR overstates quality — use NDCG or Recall@K there.

2.4 Normalized Discounted Cumulative Gain (NDCG)

MRR is binary (relevant/not) and only counts the first hit. NDCG handles graded relevance (a chunk can be perfectly, partially, or marginally relevant) and rewards good ordering across the whole list via logarithmic position discounting.

DCG@K  = Σ (relᵢ / log₂(i + 1))       for i = 1..K   (relᵢ = graded relevance of chunk at rank i)
IDCG@K = DCG of the ideal (perfectly-sorted) ranking
NDCG@K = DCG@K / IDCG@K                ∈ [0, 1]

The log₂(i+1) denominator is the positional discount: rank 1 divides by log₂2 = 1.0, rank 2 by log₂3 ≈ 1.585, rank 3 by log₂4 = 2.0. A relevant chunk contributes progressively less the deeper it sits — mirroring the LLM's declining attention. Normalizing by IDCG (the best possible ordering for that query's relevance labels) makes scores comparable across queries with different numbers of relevant chunks.

Worked example. Graded relevance labels (3 = perfect, 2 = good, 1 = marginal, 0 = irrelevant) for retrieved ranking [3, 2, 0, 1]:

DCG  = 3/log₂2 + 2/log₂3 + 0/log₂4 + 1/log₂5
     = 3/1.000 + 2/1.585 + 0/2.000 + 1/2.322
     = 3.000 + 1.262 + 0 + 0.431 = 4.693
Ideal ordering = [3, 2, 1, 0]:
IDCG = 3/1.000 + 2/1.585 + 1/2.000 + 0 = 3.000 + 1.262 + 0.500 = 4.762
NDCG = 4.693 / 4.762 = 0.985

Use NDCG as the primary retrieval metric when relevance is graded and multi-chunk synthesis matters; use MRR for single-answer factoid retrieval; use Recall@K as the hard ceiling check on end-to-end feasibility. Report all three — they answer different questions and disagreement between them is itself diagnostic (high Recall@K but low NDCG = the answer is present but poorly ranked → reranker problem).

3. The RAG Confusion Matrix

Semantic retrieval is a binary classification problem per chunk: "is this chunk relevant to the query?" The classic 2×2 confusion matrix adapts directly, and grounds F1-based tuning of chunk size and overlap.

                          ┌─────────────────────── GROUND TRUTH ───────────────────────┐
                          │   RELEVANT (in R)            │   NOT RELEVANT              │
        ┌─────────────────┼──────────────────────────────┼─────────────────────────────┤
 RETRIEVED (in top-K)     │   TRUE POSITIVE  (TP)         │   FALSE POSITIVE  (FP)      │
        │   "fetched"     │   correct chunk fetched       │   irrelevant noise fetched  │
        │                 │   ✅ signal in the window     │   ⚠️ dilutes context,       │
        │                 │                               │      wastes tokens,          │
        │                 │                               │      triggers Lost-in-Middle │
        ├─────────────────┼──────────────────────────────┼─────────────────────────────┤
 NOT RETRIEVED           │   FALSE NEGATIVE  (FN)         │   TRUE NEGATIVE  (TN)       │
        │   "not fetched" │   ❌ critical context MISSED  │   correctly ignored          │
        │                 │      → answer impossible or   │   irrelevant chunk left out  │
        │                 │        built on wrong evidence│   ✅ (the vast majority)     │
        └─────────────────┴──────────────────────────────┴─────────────────────────────┘

Precise definitions in the semantic-search context:

  • True Positive (TP) chunk: a chunk that both (a) appears in the top-K retrieved set and (b) genuinely contains information needed to answer the query. This is the only quadrant that helps the generator.
  • False Positive (FP) chunk: a chunk retrieved into the top-K that is semantically near but factually irrelevant — the classic failure of dense retrieval, where embedding similarity ("talks about the same topic") diverges from true relevance ("answers this question"). FPs are the direct cause of context dilution, token waste, and Lost-in-the-Middle burial of the real TPs.
  • False Negative (FN) chunk: a genuinely relevant chunk that exists in the vector DB but did not make the top-K — the answer-bearing evidence stranded at rank K+1, or a chunk whose embedding poorly represents its content (e.g., a table mangled at chunk time). FNs are the most dangerous quadrant because they are invisible without ground truth — the pipeline returns a confident answer with no signal that the key fact was omitted.
  • True Negative (TN) chunk: an irrelevant chunk correctly left out. In retrieval, TN is the overwhelming majority (millions of chunks, of which K are fetched), so accuracy is a useless metric here — a system that retrieves nothing scores ~99.99% "accuracy". This is precisely why Precision/Recall/F1, not accuracy, govern retrieval.

Computing F1 for chunk-size / overlap tuning:

Precision = TP / (TP + FP)      Recall = TP / (TP + FN)
F1 = 2 · (Precision · Recall) / (Precision + Recall)      (harmonic mean → punishes imbalance)

The chunking trade-off F1 exposes:

Chunk strategy Effect on FP Effect on FN Net
Large chunks (1000+ tokens) ↑ FP — each chunk carries off-topic filler, precision drops ↓ FN — more likely to contain the answer somewhere Recall ↑, Precision ↓
Small chunks (128–256 tokens) ↓ FP — tight, on-topic ↑ FN — answer split across a chunk boundary, half retrieved Precision ↑, Recall ↓
Overlap (sliding window, 10–20%) neutral ↓ FN — spans boundaries so answers aren't split Recall ↑ at storage cost

Because F1 is the harmonic mean, it collapses toward the weaker of precision/recall — a system at P=0.9, R=0.3 scores F1=0.45, not the arithmetic 0.6. This makes F1 the correct single objective for a chunk-size sweep: grid-search chunk_size × overlap, compute F1 per configuration on a labelled query set, and pick the maximum. The F1-optimal chunking is usually small chunks with modest overlap plus a reranker, capturing small-chunk precision while overlap recovers boundary-split recall.

4. Inference-Layer & LLM-as-a-Judge Metrics (The "G" Surface)

Retrieval metrics need labelled relevant-chunk sets. Generation metrics evaluate the free-text answer — where exact-match is impossible — using a judge LLM to decompose and score. This is the RAG Triad (Ragas/TruLens): Faithfulness, Answer Relevance, Context Relevance, forming a closed loop over Query → Context → Answer.

                    ┌──────────── the RAG Triad ────────────┐
                    │                                        │
        Context Relevance                          Answer Relevance
        (context ↔ query)                          (answer ↔ query)
                    │              QUERY                     │
                    │             /     \                    │
                    │        CONTEXT ─── ANSWER              │
                    └──────── Faithfulness (answer ↔ context)┘

4.1 Faithfulness / Groundedness

Definition: the fraction of factual claims in the generated answer that are directly supported by (entailed by) the retrieved context. Range [0, 1]. This is the primary anti-hallucination metric — it detects the Scenario-B failure where the generator answers from parametric memory instead of the provided context.

Step-by-step execution logic:

  1. Claim extraction. A judge LLM is prompted to decompose the generated answer into an enumerated set of atomic factual statements — each a single, independently-verifiable proposition. "Acme was founded in 1997 in Berlin and makes turbines" → {c1: founded 1997, c2: founded in Berlin, c3: makes turbines}. Atomicity matters: a compound sentence scored as one claim hides partial hallucination.
  2. Per-claim entailment (NLI). For each claim cᵢ, the judge is given the retrieved context and asked a binary/graded entailment question: "Is cᵢ supported by this context?" → verdict ∈ {supported, not-supported} (some implementations add "contradicted"). This is a Natural Language Inference task performed by the judge.
  3. Aggregation. Faithfulness = (# supported claims) / (# total claims). Answer with claims {c1 ✅, c2 ❌ (context said Munich), c3 ✅} → 2/3 = 0.67.

Edge cases & failure modes of the metric itself: - Claim granularity sensitivity — over-splitting inflates the denominator; under-splitting hides errors. Pin the extraction prompt and judge version, because a judge upgrade silently re-scales the metric. - Numerical/temporal claims — judges are weak at "founded 1997" vs "founded in the late 90s"; supply tolerance rules or the metric produces false hallucination flags. - Multi-hop entailment — a claim supported only by combining two context chunks may be scored "not supported" if the judge checks chunks independently. - Faithfulness ≠ correctness. A faithful answer grounded in a wrong retrieved chunk (Scenario C) scores faithfulness = 1.0 while being factually false. Faithfulness measures grounding, not truth — which is exactly why it must be paired with retrieval metrics.

4.2 Answer Relevance

Definition: how well the answer addresses the user's actual question, independent of retrieval — penalizing incompleteness, evasiveness, and off-topic drift. It catches answers that are faithful and fluent but don't actually respond to what was asked.

Execution logic (the reverse-question technique used by Ragas):

  1. Feed the generated answer to a judge LLM and ask it to generate N synthetic questions that this answer would be a good response to.
  2. Embed each generated question and the original user query.
  3. Compute cosine similarity between the original query and each synthetic question; average them: Answer Relevance = (1/N) · Σ cos(q_original, q_generated_i).

Interpretation & mechanics: if the answer is on-point, the questions it "answers" cluster tightly around the original query (high cosine, → 1.0). If the answer drifted, hedged ("I don't have enough information, but generally…"), or padded with tangents, the reverse-generated questions scatter away from the original (low cosine). It measures semantic alignment between intent and response without needing a ground-truth answer. Edge cases: noncommittal answers score low (correctly); overly broad answers that technically touch the query score deceptively high (pair with a completeness check); the metric is sensitive to the embedding model used for the cosine step — pin it alongside the judge.

4.3 Context Recall & Context Precision

These evaluate the retrieved context against a ground-truth reference answer — the bridge metrics that connect the R-surface to the G-surface, requiring a labelled {query, ground_truth_answer, [relevant_chunks]} dataset.

Context Recall — did retrieval fetch all the facts the ground-truth answer needs? - The judge decomposes the ground-truth answer into atomic claims. - For each claim, it checks whether that claim is attributable to the retrieved context. - Context Recall = (# ground-truth claims supported by retrieved context) / (# ground-truth claims). - A low score means the answer-bearing evidence never reached the window — a pure retrieval recall failure (FN quadrant). This is the metric that catches Scenario C/D at the context layer.

Context Precision — are the relevant chunks ranked ahead of the irrelevant ones? - For each retrieved chunk at rank k, the judge labels it relevant/irrelevant to the ground-truth answer. - The metric computes a rank-weighted precision (precision@k averaged over the positions of relevant items), rewarding systems that place relevant chunks high: Context Precision = Σ (Precision@k · relevantₖ) / (total relevant chunks). - Low Context Precision with high Context Recall = the right chunks were fetched but buried among noise → a reranking problem, and a Lost-in-the-Middle risk. This is the judge-based analogue of NDCG/MRR.

How the four generation-side metrics localize failure together:

Faithfulness Answer Relevance Context Recall Context Precision Diagnosis
high high high high Healthy pipeline
low high high high Generator hallucinating despite good context → G-fix (prompt/model)
high low high high Faithful but not answering the question → prompt/instruction fix
high high low — Evidence never retrieved → R-fix (chunking/embedding/K)
high high high low Right evidence, wrong order → reranker fix

5. System Latency, Throughput & Stress Testing

Quality metrics are necessary but not sufficient — a faithful answer delivered in 8 seconds fails the product. RAG latency decomposes into three additive deltas on the critical path, each with distinct scaling behaviour and test strategy.

   T_total  =  ΔTₑ (embed query)  +  ΔTₛ (vector search)  +  [ΔT_rerank]  +  ΔTg (LLM generation)
              ├── ~5–50 ms ──┤   ├── 1–100 ms, scales ─┤                ├── 500 ms–10 s, dominant ─┤
                (fixed-ish)        with index size N        (optional)      (scales with output tokens)

ΔTₑ — Embedding delta (query encoding). Time to convert the incoming query string into a vector. Roughly constant per query (independent of corpus size) — it depends on the embedding model size, sequence length, and whether it runs locally vs. via a network API call. A remote embedding API adds network RTT and is a hidden tail-latency source (test p99, not just mean). Test: isolate by embedding a fixed query 1000× and recording the distribution; a bimodal distribution reveals cold-start/model-reload effects.

ΔTₛ — Vector search delta (index navigation). Time for the ANN index to return top-K. This is where corpus population size N bites. Exact (flat) search is O(N·d) and degrades linearly — fine at 100k, unusable at 10M. Production uses approximate indexes: - HNSW (Hierarchical Navigable Small World) — a multi-layer proximity graph; search is O(log N) hops, giving near-flat latency as N grows, at the cost of recall (tunable via efSearch) and high memory. The key test axis is the recall/latency curve as efSearch varies — higher ef = better recall, more hops, more latency. - IVF (Inverted File) — partitions vectors into nlist clusters; search probes nprobe clusters. Latency scales with nprobe, recall with the ratio nprobe/nlist.

The critical stress test — QPS under varying N: the same index has different latency at 100k vs. 10M vectors even with HNSW, because graph depth, memory pressure, and cache-miss rates rise. Measure ΔTₛ as a function of N (populate the index at 100k, 1M, 10M and re-benchmark) — a linearly-rising ΔTₛ betrays a misconfigured ANN falling back toward brute force.

ΔTg — Generation delta (token output). Almost always the dominant term. Scales with output token count (each token is a forward pass) and, for streaming, is characterized by Time-To-First-Token (TTFT) and inter-token latency. Longer retrieved context also inflates the prefill stage (processing input tokens) — a second reason oversized K hurts. Test: report TTFT (UX-critical) separately from total generation time; assert both against SLA.

Effective QPS vs. Input QPS — the saturation test. Input QPS is the request rate you offer the system; Effective QPS is the rate it actually completes within the latency SLA.

   Input QPS ──▶ [ system ] ──▶ Effective QPS

   Effective QPS
      ▲
      │           ,-----•------  ← saturation plateau (throughput ceiling)
      │        ,-'      |  ↘ latency climbs, some requests breach SLA
      │      ,'         |     ↘ effective QPS may DROP (queue collapse)
      │    ,'           |
      │  ,'   (linear: system keeps up)
      └──────────────────────────▶ Input QPS
                    knee = max sustainable throughput

Below the knee, Effective ≈ Input (system keeps up). At the knee, requests queue, latency crosses the SLA, and Effective QPS plateaus then can collapse (queue thrashing, timeouts, retries amplifying load). Test protocol: ramp Input QPS while recording Effective QPS and p50/p95/p99 latency; the max sustainable throughput is the Input QPS at which p99 first breaches the SLA — not the point of maximum raw throughput. Run this at each corpus size (100k / 1M / 10M), because the knee moves left as N grows (ΔTₛ rises), and the generation tier (GPU-bound ΔTg) and retrieval tier (memory/CPU-bound ΔTₛ) saturate at different input rates — identify which tier is the bottleneck before scaling the wrong one.

6. Consolidated Metric Reference

Metric Surface Formula (core) Answers Pair with
Cosine similarity R q·c / (‖q‖‖c‖) Semantic closeness Self-retrieval sanity test
Precision@K R TP / K Signal-to-noise in window Recall@K
Recall@K R TP / \|R\| Feasibility ceiling Precision@K
MRR R mean(1/rank_first) First-hit ranking NDCG
NDCG@K R DCG/IDCG Graded-relevance ordering MRR
F1 R 2PR/(P+R) Chunk-tuning objective Confusion matrix
Faithfulness G supported/total claims Hallucination Context Recall
Answer Relevance G mean cos(q, q_gen) On-topic / complete Faithfulness
Context Recall R↔G gt-claims supported/total Evidence completeness Context Precision
Context Precision R↔G rank-weighted precision Evidence ordering NDCG
ΔTₑ/ΔTₛ/ΔTg Perf additive critical path Latency budget Effective QPS

The golden rule of RAG evaluation: never report a single end-to-end score. Always decompose into R-surface (IR metrics), G-surface (Triad metrics), and the R↔G bridge (Context Recall/Precision), because the fix for a wrong answer depends entirely on which surface failed — and the symptoms are identical from the outside.

Where to Go Next