Skip to content

Cloud LLM Evaluation Tools — Bedrock, Azure AI Foundry & Vertex AI

What this is: how the three major clouds let you evaluate LLM quality as a managed service — Amazon Bedrock Evaluations, Azure AI Foundry evaluation, and Google Vertex AI's Gen AI Evaluation Service — and how they compare to open-source frameworks like Ragas and DeepEval. Written from the QA/evaluation angle: what each service measures, how it fits a CI pipeline, and when to choose cloud vs open source.

Prerequisites: LLM & Agent Evaluation Matrix (the metrics), Enterprise LLM Platforms (the platforms themselves).

1. Why Cloud Evaluation Services Exist

Open-source eval frameworks (Ragas, DeepEval, promptfoo) run your code against your judge model. The clouds productised the same idea as a managed service:

  • No harness to build — upload a prompt dataset, pick metrics, get a report.
  • Curated judge models & rubrics — the LLM-as-judge prompts are pre-engineered and calibrated.
  • Governance built in — results land in the same console as your deployment, with IAM, audit trails, and region controls (matters in regulated environments).
  • Human-review workflows — managed UIs for SME review teams, not just automated scoring.
flowchart LR
    DS["📋 Your dataset<br/>(prompts + optional references)"] --> SVC["☁️ Managed evaluation job<br/>Bedrock / AI Foundry / Vertex"]
    TGT["🤖 Target: hosted model,<br/>RAG pipeline, or<br/>bring-your-own responses"] --> SVC
    SVC --> J["⚖️ Judge model + curated rubrics<br/>(LLM-as-a-judge)"]
    J --> REP["📊 Scored report in console<br/>+ exportable results"]
    REP --> GATE["🚦 CI/CD threshold gate"]

2. Amazon Bedrock Evaluations

Bedrock's evaluation service (LLM-as-a-judge generally available since 2025) evaluates models and full RAG systems (Knowledge Bases) with automated, judge-based, and human-based modes.

Mode What it does
Automatic (programmatic) Classic NLP metrics (accuracy, robustness, toxicity) on built-in or custom datasets
LLM-as-a-judge A curated judge model scores outputs on quality metrics — correctness, completeness, style/tone — and responsible-AI metrics — harmfulness, answer refusal
Human evaluation Managed workflow for your own SME review team with custom metrics
RAG evaluation Evaluates retrieve-and-generate quality for Knowledge Bases (context relevance, groundedness-style checks)

QA-relevant details:

  • Bring your own inference responses — you can evaluate outputs from any model or system hosted anywhere by including the responses in your input dataset. Your system under test does not have to run on Bedrock.
  • Datasets are JSONL in S3; jobs run via console or boto3/CLI — scriptable into CI.
  • Judge-based jobs cost a fraction of human review (AWS cites large cost reductions vs human evaluation at scale).
# CI-friendly: kick off a Bedrock LLM-as-judge evaluation job
import boto3
bedrock = boto3.client("bedrock")
job = bedrock.create_evaluation_job(
    jobName="rag-regression-2026-07",
    evaluationConfig={"automated": {
        "datasetMetricConfigs": [{
            "taskType": "QuestionAndAnswer",
            "dataset": {"name": "golden-set",
                        "datasetLocation": {"s3Uri": "s3://evals/golden.jsonl"}},
            "metricNames": ["Builtin.Correctness", "Builtin.Completeness",
                            "Builtin.Harmfulness"],
        }],
        "evaluatorModelConfig": {"bedrockEvaluatorModels": [
            {"modelIdentifier": "anthropic.claude-sonnet"}]},   # the judge
    }},
    inferenceConfig={"models": [{"bedrockModel":
        {"modelIdentifier": "my-app-model"}}]},
    outputDataConfig={"s3Uri": "s3://evals/results/"},
    roleArn="arn:aws:iam::123456789:role/BedrockEvalRole",
)

3. Azure AI Foundry Evaluation

Azure AI Foundry (the evolution of Azure AI Studio) bakes evaluation into the project workflow: evaluate in the playground, in bulk runs, or locally in code via the azure-ai-evaluation SDK — then track results in the portal.

Built-in evaluator families:

Family Example evaluators
RAG / quality (AI-assisted) Groundedness, Relevance, Coherence, Fluency, Retrieval quality
Similarity / classic Similarity, F1, BLEU/ROUGE-style comparisons to a reference
Risk & safety Violence, sexual, self-harm, hate/unfairness, protected-material, jailbreak detection
Agentic (newer) Intent resolution, tool-call accuracy, task adherence
Custom Your own prompt-based or code-based evaluators 
# azure-ai-evaluation SDK — run evaluators locally / in CI
from azure.ai.evaluation import evaluate, GroundednessEvaluator, RelevanceEvaluator

model_config = {"azure_endpoint": ENDPOINT, "azure_deployment": "gpt-judge"}

result = evaluate(
    data="golden_set.jsonl",             # query, context, response columns
    evaluators={
        "groundedness": GroundednessEvaluator(model_config),
        "relevance":    RelevanceEvaluator(model_config),
    },
)
assert result["metrics"]["groundedness.mean"] >= 4.0   # CI threshold gate

QA-relevant details:

  • The same evaluators run pre-production (bulk eval on datasets) and in production (continuous evaluation / monitoring) — one metric vocabulary across the lifecycle.
  • Safety evaluators use Azure's content-safety stack — useful for regulated sign-off evidence.
  • Results are versioned per run in the Foundry portal — an audit trail for "quality at release" questions.

4. Google Vertex AI — Gen AI Evaluation Service

Vertex AI's evaluation service centres on judge-model scoring with pointwise and pairwise modes, tightly integrated with the Vertex SDK.

Concept Meaning
Pointwise Judge scores one model's output against criteria (e.g. groundedness 1–5)
Pairwise Judge compares two models/prompts head-to-head and picks a winner — ideal for "is the new prompt actually better?" A/B decisions
Bias controls Response flipping and multi-sampling reduce judge position/order bias
Metric types Model-based (fluency, coherence, safety, groundedness, instruction following) + computation-based (BLEU, ROUGE, exact match) + custom rubrics 
# Vertex AI rapid evaluation — pairwise prompt comparison
from vertexai.evaluation import EvalTask, PairwiseMetric

eval_task = EvalTask(
    dataset=golden_df,                       # prompts + responses A/B
    metrics=[PairwiseMetric(metric="pairwise_summarization_quality")],
)
result = eval_task.evaluate()
# result tells you the win-rate of candidate B over baseline A

QA-relevant details: pairwise mode is the cleanest managed answer to prompt-regression testing — "did this prompt change make answers better or worse?" — which is awkward to do rigorously by hand.

5. Comparison & Choosing

Bedrock Evaluations Azure AI Foundry Vertex AI Eval
LLM-as-judge ✅ GA, curated judges ✅ AI-assisted evaluators ✅ default judge, pointwise + pairwise
RAG-specific eval ✅ Knowledge Bases eval ✅ Groundedness/Retrieval evaluators ✅ grounding metrics
Safety metrics ✅ harmfulness, refusal ✅ full content-safety family + jailbreak ✅ safety criteria
Human review workflow ✅ managed ✅ portal-based ✅ via labelling services
Evaluate external systems ✅ bring-your-own responses ✅ SDK runs anywhere ✅ SDK-based, bring your data
Local/CI SDK boto3 (job-based) azure-ai-evaluation (runs locally) Vertex SDK
Standout RAG + BYO responses + cost-efficient judge one metric vocabulary dev→prod + safety depth pairwise A/B with bias controls

Cloud service vs open source (Ragas / DeepEval)

Choose cloud eval when… Choose open source when…
Your stack already lives on that cloud (IAM, audit, data residency) You need portability across providers
You need managed human-review workflows You want full control of judge prompts & metric logic
Governance/audit evidence matters (regulated releases) You want free/cheap local runs in every PR
You want zero eval-harness maintenance You need custom metrics beyond the built-ins

Pragmatic pattern: open-source evals (fast, free, per-PR) as the inner loop; a cloud evaluation job as the release gate — the versioned, auditable "quality at sign-off" record. They complement rather than compete.

6. CI/CD Integration Pattern

Regardless of vendor, the pipeline shape is identical to any LLM eval gating:

flowchart LR
    PR["PR: prompt / model / RAG change"] --> FAST["⚡ Inner loop<br/>Ragas / DeepEval on 50-case subset"]
    FAST -->|pass| FULL["☁️ Cloud eval job<br/>full golden set + safety metrics"]
    FULL --> GATE{"thresholds met?<br/>e.g. groundedness ≥ 4.0<br/>harmfulness = 0"}
    GATE -->|yes| SHIP["✅ merge / deploy<br/>report archived as evidence"]
    GATE -->|no| FIX["❌ block + scored failures<br/>back to author"]

Checklist for any of the three services:

  • Golden dataset versioned in repo / S3 / blob — not ad hoc
  • Judge model pinned (a judge upgrade is an environment change — re-baseline)
  • Thresholds explicit and reviewed like code
  • Results exported and archived per release (audit trail)
  • Safety metrics always on, not only quality metrics

Where to Go Next