Skip to content

Testing Deep Dives — Confusion Matrix, SageMaker, Agent Testing & Telemetry

What this is: four focused deep dives that come up constantly in AI/ML QA work and interviews — the confusion matrix (classic-ML evaluation), Amazon SageMaker (the ML platform and how to test on it), agent testing (validating tool-using AI), and telemetry analysis (reading production signals to find problems). Each links to the broader guides it extends.

1. The Confusion Matrix — Classic-ML Evaluation

Before generative AI, model quality was measured with a confusion matrix: a table comparing predicted labels against true labels. It is still how you test any classification model — fraud scoring, spam detection, churn, sentiment — and the foundation the RAG confusion matrix is built on.

1.1 The 2×2 (binary classification)

                        │        ACTUAL (ground truth)         │
                        │   Positive        │   Negative       │
   ─────────────────────┼───────────────────┼──────────────────┤
   PREDICTED  Positive  │  True Positive    │  False Positive  │
                        │       (TP)        │   (FP, Type I)   │
   ─────────────────────┼───────────────────┼──────────────────┤
   PREDICTED  Negative  │  False Negative   │  True Negative   │
                        │   (FN, Type II)   │       (TN)       │
  • TP — predicted positive, actually positive (fraud flagged, and it was fraud).
  • TN — predicted negative, actually negative (legit approved, and it was legit).
  • FP (Type I error) — false alarm (legit transaction blocked as fraud).
  • FN (Type II error) — miss (fraud approved as legit — usually the costliest).

1.2 The metrics derived from it

Metric Formula Answers Watch out
Accuracy (TP+TN) / total Overall % correct Misleads on imbalanced data — 99% "accuracy" by always predicting "not fraud"
Precision TP / (TP+FP) Of flagged positives, how many were real? Low precision = too many false alarms
Recall / Sensitivity TP / (TP+FN) Of real positives, how many did we catch? Low recall = dangerous misses
Specificity TN / (TN+FP) Of real negatives, how many correctly cleared? —
F1 score 2·(P·R)/(P+R) Harmonic mean of precision & recall The go-to single number for imbalanced classes

1.3 The precision–recall trade-off & thresholds

A classifier outputs a probability; a threshold turns it into a label. Moving the threshold trades precision against recall:

  raise threshold (e.g. 0.9) ─▶ fewer positives ─▶ ↑ precision, ↓ recall  (cautious)
  lower threshold (e.g. 0.3) ─▶ more positives  ─▶ ↓ precision, ↑ recall  (aggressive)
  • Fraud/medical: favour recall — a miss (FN) is catastrophic, a false alarm (FP) is just an inconvenience.
  • Spam/content moderation: often favour precision — blocking a real email (FP) annoys users more than letting one spam through.
  • ROC curve / AUC — plots TPR vs FPR across all thresholds; AUC (area under curve, 0.5 = random, 1.0 = perfect) summarises threshold-independent quality. For imbalanced data prefer the Precision–Recall curve / PR-AUC.

1.4 Multi-class

For N classes it's an N×N matrix; the diagonal is correct predictions, off-diagonal cells show which class got confused with which — invaluable for spotting, e.g., "the model keeps calling class 3 a class 5". Compute per-class precision/recall, then macro-average (treat classes equally) or weighted-average (by class frequency).

1.5 QA use

Your job is to assert the model beats a baseline on a held-out test set: fix a threshold, compute the matrix, gate on F1 ≥ target and recall ≥ floor. This is the classic-ML half of a hybrid system — see From ML to Generative AI for how it sits beside LLM evaluation.

2. Amazon SageMaker — The ML Platform (and How to Test It)

Where Bedrock consumes foundation models, SageMaker builds and operates your own models end to end. For QA, SageMaker matters both as the thing under test and as a toolbox of built-in test capabilities.

2.1 The lifecycle SageMaker covers

flowchart LR
    D["Data<br/>(S3)"] --> P["Prepare<br/>Data Wrangler / Feature Store"]
    P --> T["Train & Tune<br/>managed compute + HPO"]
    T --> R["Model Registry<br/>version + approve"]
    R --> E["Deploy Endpoint<br/>real-time / serverless / batch"]
    E --> M["Monitor<br/>Model Monitor + Clarify"]
    M -.->|drift detected| T

2.2 SageMaker features a QA engineer actually uses

Feature What it gives QA
Model Registry Versioned models + approval gates — test the right version, block unapproved ones
SageMaker Clarify Bias detection (pre-train data bias + post-train prediction bias) and explainability (SHAP feature attributions)
Model Monitor Production drift detection — data-quality drift, model-quality drift, bias drift, feature-attribution drift
Endpoints The deployable unit you load-test and functionally test (real-time / serverless / batch)
Batch Transform Score a whole test dataset offline → build the confusion matrix (§1)
Pipelines MLOps CI/CD — attach test steps as pipeline stages

2.3 How to test a SageMaker model / endpoint

  • Offline evaluation — run Batch Transform on a labelled hold-out set, compute confusion-matrix metrics (§1), gate the pipeline on thresholds.
  • Endpoint functional tests — invoke the endpoint API: correct input schema accepted, malformed input rejected cleanly, output shape/range valid, latency within SLA.
  • Load / concurrency — hit the endpoint at target TPS; assert auto-scaling holds latency.
  • A/B & shadow testing — SageMaker supports production variants: route a % of traffic to a new model (A/B) or mirror traffic to a shadow model without affecting users, then compare metrics before promoting.
  • Drift monitoring as continuous testing — Model Monitor emits telemetry (§4); alert when input distributions or accuracy proxies drift beyond a baseline → triggers retraining.
  • Bias & fairness gate — run Clarify; assert disparity metrics across protected groups stay within tolerance.

Deeper platform context: AI on AWS. SageMaker is the "build your own" pillar; Bedrock is the "use managed models" pillar.

3. Agent Testing — Validating Tool-Using AI

An agent is an LLM that plans, calls tools, observes results, and loops until a task is done. You stop testing one answer and start testing a trajectory. This section extends AI Test Strategy §3 with the how.

3.1 What breaks in agents

Failure Example
Wrong tool selected Calls delete_record when asked to read one
Malformed / hallucinated tool call Invents a tool, or emits args that don't match the schema
Infinite / runaway loop Re-plans forever, burning tokens and cost
Ignored tool error Tool returns 500; agent proceeds as if it succeeded
Wrong task "completed" Confidently reports done, but did the wrong thing
Unsafe action Sends an email / spends money without confirmation
Injection via tool result Malicious text in retrieved data hijacks the plan

3.2 Test at three altitudes

  ┌ Tool-call level ─ every emitted call: right tool? args valid JSON? schema-valid? no hallucinated tool?
  ├ Trajectory level ─ sensible step order · no loops (step budget) · errors handled · state carried across steps
  └ Outcome level ─── task actually done? side-effects correct (record exists)? judge scores final result vs goal

result = run_agent("Find overdue invoices and draft reminders")

# tool-call level
for call in result.tool_calls:
    assert call.name in REGISTERED_TOOLS              # no hallucinated tool
    validate(json.loads(call.arguments), SCHEMAS[call.name])   # schema-valid args

# trajectory level
assert result.steps <= MAX_STEPS                      # no runaway loop
assert "query_invoices" in result.tools_used          # right tool selected

# outcome level
assert db.count_draft_reminders() == db.count_overdue()   # side effect correct
score = judge(goal="draft reminders for overdue invoices", transcript=result.transcript)
assert score["task_success"] >= 4

3.3 Metrics & techniques specific to agents

  • Task success rate — % of golden tasks completed correctly end-to-end.
  • Tool-selection accuracy — right tool chosen for the step.
  • Step efficiency — steps taken vs optimal (detects dithering/loops).
  • Cost & latency per task — budget enforcement asserted in tests.
  • Mock the tools — in tests, replace real tools with deterministic mocks so you isolate the agent's reasoning from flaky external APIs.
  • Adversarial — inject malicious content into tool results; assert the plan isn't hijacked (prompt injection).
  • Multi-agent adds hand-off contract tests + chaos injection (AI Test Strategy §5).

4. Telemetry Analysis — Reading Production Signals

Telemetry is the data a running system emits about itself. Analysis is turning that stream into answers: is it healthy, what's it costing, and where is it failing? This extends the LiteLLM & Langfuse observability guide toward the analysis discipline.

4.1 The three pillars of observability

Pillar What it is AI-system example
Metrics Numeric time-series, aggregatable Requests/sec, p95 latency, tokens/min, $ cost, error rate, quality score
Logs Timestamped event records The exact prompt, the completion, an error stack
Traces One request's path across services (spans) UI → gateway → vector search → LLM, each span timed

The glue is a correlation ID (trace_id) stamped across every service, so a metric spike → the logs → the exact trace can be followed in one hop (AI Rollout Part 3).

4.2 What to measure for AI systems specifically

  Performance   TTFT · inter-token latency · total latency (p50/p95/p99)
  Cost          tokens in/out · $ per request · $ per user/team · budget burn
  Reliability   error rate · timeout rate · fallback-fired rate · rate-limit (429) rate
  Quality       eval scores (faithfulness, relevance) sampled in prod · thumbs up/down
  Drift         input distribution shift · output length/scores over time

4.3 How to analyse telemetry to find problems

  • RED method (for request-driven services): track Rate, Errors, Duration — three dashboards catch most incidents.
  • USE method (for resources): Utilisation, Saturation, Errors — for GPUs/queues behind inference.
  • Percentiles, not averages — the mean hides tail pain; a p99 latency spike is the user experience that churns. Always chart p50/p95/p99.
  • Correlate by dimension — group cost/errors by model, user, team, prompt version → the spike localises to a culprit (a runaway user, a bad prompt release).
  • Anomaly detection & alerting — set SLO-based alerts (e.g. "p95 > 3s for 5 min", "error rate > 2%", "faithfulness score < 0.8"); page before users notice.
  • Trace-level root cause — a slow/failed request → open its trace → see which span (retrieval? generation? a tool?) blew the budget.
  • Regression from telemetry — a production failure trace becomes a new golden test case; feed it back into CI (AI Test Strategy).

4.4 The toolchain

Layer Tools
Standard/instrumentation OpenTelemetry (vendor-neutral traces/metrics/logs)
Metrics & dashboards Prometheus + Grafana, CloudWatch, Datadog
LLM-specific tracing Langfuse, LangSmith, Arize/Phoenix
Log aggregation Splunk, ELK, CloudWatch Logs

QA framing: telemetry is production testing. SLOs are your production assertions; alerts are your failing tests; traces are your evidence. The eval scores you write back into telemetry (cloud eval services) turn monitoring into continuous quality measurement.

Where to Go Next