Skip to content

Enterprise LLM Gateway Architecture — Proxy → Bedrock → Claude

What this is: an end-to-end walkthrough of the architecture most enterprises use to give developers and employees safe, governed access to LLMs. A self-hosted proxy (gateway) running in the organization's own cloud sits between users and the model, routes each request to the right LLM on Amazon Bedrock (e.g., Claude), and uses PostgreSQL for keys, budgets, cost, and logs.

Companion pages: AI Rollout — Part 1 §5 (Inference Gateway), LLM Observability — LiteLLM & Langfuse, Enterprise LLM Platforms.

1. The Big Picture

The key idea: there are two different "clouds" here, and they are not the same kind of thing.

  • Your cloud (self-managed): the proxy/gateway software and its PostgreSQL database are software you install and run inside your own AWS/Azure/on-prem account. You own the servers.
  • Amazon Bedrock (fully managed): you don't install anything. Bedrock is a serverless, managed inference API — AWS runs Anthropic's Claude on its own infrastructure. You just get an API endpoint + IAM permissions. This is why "it's not a usual cloud" — you never provision a Claude server; you call an API and AWS executes the model for you, inside your chosen region.
flowchart LR
    subgraph CLIENTS["👥 Callers (client side)"]
        DEV["👨‍💻 Developer systems<br/>IDE, scripts, CI"]
        APP["🧩 Internal apps<br/>chatbot, RAG service"]
        EMP["🧑‍💼 Employees<br/>internal AI portal"]
    end

    subgraph ORG["🏢 YOUR CLOUD / VPC  (software you install & run)"]
        GW["🚪 LLM Gateway / Proxy<br/>(e.g. LiteLLM Proxy)<br/>routing · virtual keys · budgets · guardrails"]
        PG[("🗄️ PostgreSQL<br/>keys · teams · budgets<br/>spend · request logs · config")]
        GW <--> PG
    end

    subgraph AWS["☁️ Amazon Bedrock  (AWS-managed, serverless)"]
        BR["🛡️ Bedrock API endpoint<br/>(regional, IAM-authed)"]
        CLAUDE["🤖 Claude (Anthropic)<br/>runs inside Bedrock"]
        BR --> CLAUDE
    end

    DEV & APP & EMP -->|"OpenAI-format request<br/>+ virtual key"| GW
    GW -->|"private link / VPC endpoint<br/>IAM-signed request"| BR
    CLAUDE -->|"completion"| BR -->|"response + token usage"| GW
    GW -->|"answer + cost logged"| DEV & APP & EMP
    GW -.->|"optional: traces/logs"| OBS["📊 Observability<br/>(Langfuse / OTEL)"]

2. Where Is Each Piece Actually Installed?

This is the crux of the question — what do you run, and what does AWS run?

Component Who runs it Where it lives You install it?
Client apps / dev systems You Laptops, CI, app servers — (they just call the gateway)
LLM Gateway / Proxy You A container in your cloud (ECS / EKS / EC2 / Azure / on-prem K8s) ✅ Yes — you deploy the software
PostgreSQL You Your cloud (AWS RDS, Azure DB, or self-managed) ✅ Yes — you provision the DB
Bedrock API endpoint AWS AWS's regional infrastructure ❌ No — managed service, you get IAM access
Claude model AWS (Anthropic's model) Inside Bedrock, never in your VPC ❌ No — you never host the model
Observability (Langfuse) You (or Langfuse Cloud) Your cloud or SaaS Optional 
   ┌──────────────── YOU OWN & OPERATE ────────────────┐   ┌──── AWS OPERATES ────┐
   │  Gateway/Proxy container  +  PostgreSQL database    │   │  Bedrock API + Claude │
   │  (your VPC, your compute, your IAM role)            │   │  (managed, serverless)│
   └────────────────────────────────────────────────────┘   └───────────────────────┘
                         │  private, IAM-signed call over PrivateLink / VPC endpoint  │
                         └───────────────────────────────────────────────────────────┘

Mental model: you install and run the control plane (gateway + Postgres). AWS runs the model plane (Bedrock + Claude). They connect over a private, authenticated network path — your data never touches the public internet if you use a VPC endpoint / PrivateLink.

3. The Request/Response Flow — Step by Step

Follow a single "Summarize this contract" request from a developer's app to Claude and back:

 1  Client ──▶ Gateway     : POST /chat/completions  {model:"claude", messages:[...]}
                             Header: Authorization: Bearer sk-team-legal-key   (VIRTUAL key)
 2  Gateway  ↔ PostgreSQL  : validate virtual key → look up team, budget, rate limit, model alias
 3  Gateway               : guardrails — PII scan, prompt-injection check, context-length check
 4  Gateway               : resolve alias "claude" → bedrock/anthropic.claude-...  ; pick region
 5  Gateway ──▶ Bedrock    : IAM-signed (SigV4) request over VPC endpoint / PrivateLink
 6  Bedrock ──▶ Claude     : runs inference inside AWS (in-region, optional Zero Data Retention)
 7  Claude ──▶ Bedrock ──▶ Gateway : completion + token usage (prompt/completion tokens)
 8  Gateway  ↔ PostgreSQL  : compute $ cost from tokens → write spend + request log row
 9  Gateway  ─▶ Observability : (optional) emit trace: prompt, model used, tokens, latency
10  Gateway ──▶ Client      : return the answer (OpenAI-format response)

Two directions, one door: every request goes out through the gateway and every response comes back through it. Nothing calls Bedrock directly — that single choke point is what makes central governance, cost control, and logging possible.

4. Every Element Explained

4.1 Client / Developer & User Systems

The callers: internal apps, RAG services, IDE assistants, CI jobs, and employee-facing AI portals. Critically, they never hold real cloud/provider credentials and never talk to Bedrock directly. They authenticate to the gateway with a virtual key and speak a single, standard request format (usually OpenAI-compatible), so switching the underlying model requires no client code change.

4.2 The LLM Gateway / Proxy (the "proxy" — the heart of it)

Software you install in your own cloud (commonly the open-source LiteLLM Proxy, or a managed equivalent). It is the central control plane and does six jobs:

Job What it does
Unified API One OpenAI-format endpoint for 100+ models; apps call model="claude" regardless of provider
Routing & aliasing Maps an alias (claude) to a concrete Bedrock model + region; can re-point aliases with zero downtime
Virtual keys Issues per-team/per-user keys with their own limits — real Bedrock/IAM creds stay server-side only
Budgets & rate limits Enforces spend caps and RPM/TPM per key/team; returns HTTP 429 when exceeded
Guardrails PII redaction, prompt-injection filtering, context-window boundary checks before the call leaves
Metering & logging Counts tokens, computes cost, writes every call to PostgreSQL and (optionally) to an observability sink

4.3 PostgreSQL (the state store)

The gateway is largely stateless compute; PostgreSQL is where its memory lives. It stores:

  • Virtual keys, users, teams, and their permissions — who can call what.
  • Budgets & spend — running cost per key/team/user, so limits can be enforced and reported.
  • Request/response logs & metadata — model used, token counts, latency, timestamps, cost per call (metadata-only logging is the privacy-safe default; raw prompt text is often omitted or masked).
  • Configuration — model list, routing rules, aliases (can be DB-backed rather than file-based).

Run it as a managed database (AWS RDS / Aurora, Azure Database for PostgreSQL) for backups, HA, and encryption at rest. If Postgres is down, the gateway can't validate keys or record spend — so it's a first-class production dependency, not an afterthought.

The link from your gateway to Bedrock. Using an AWS PrivateLink / VPC interface endpoint keeps traffic on AWS's private backbone — the request never traverses the public internet. Requests are IAM-signed (SigV4) using the gateway's IAM role, so Bedrock authorizes the call without any long-lived API key sitting in client apps.

4.5 Amazon Bedrock (the managed model plane)

A fully managed, serverless inference service — the "not a usual cloud" part. You don't run servers or load model weights; you call the Bedrock API and AWS executes the model. Enterprise-relevant properties:

  • In-region execution — the model runs in the AWS region you target (e.g., eu-central-1), supporting data-residency requirements.
  • Zero Data Retention (ZDR) — configurable so prompts/outputs exist only in memory during inference and aren't stored by the provider.
  • No training on your data — enterprise Bedrock terms guarantee your prompts/outputs are not used to train the foundation models.
  • IAM-native — access is governed by AWS IAM policies, not shared API keys.

4.6 Claude on Bedrock (the model)

Anthropic's Claude, served through Bedrock rather than called directly from Anthropic. Same model family, but wrapped in AWS's security, IAM, regional, and billing envelope — which is exactly why enterprises prefer it over the public API. To your gateway it's just another Bedrock modelId; to your apps it's just the alias claude.

The gateway can forward a copy of every call to a tracing platform (Langfuse, or any OTEL backend) — full trace tree, token/cost dashboards, error and latency views. This is how you see problems; see the LiteLLM & Langfuse guide.

5. Why This Architecture (the "why" behind each choice)

Design choice Problem it solves
Central proxy, not direct calls One place for auth, cost control, guardrails, and logging — no shadow AI usage
Virtual keys Developers never hold real cloud credentials; revoke/rotate per team instantly
Bedrock instead of public API IAM, in-region residency, ZDR, no-training terms → passes enterprise compliance
PrivateLink Data never hits the public internet → data-sovereignty and security sign-off
PostgreSQL Durable budgets, spend attribution, and audit logs → finance and audit requirements
Model aliasing Swap claude v1→v2 or fail over to another model with zero app changes

6. How This Maps to the QA Test Matrix

Every element above is a test target — this is the architecture your Inference Gateway test matrix exercises:

  • Gateway → high-concurrency load, failover, rate-limit (429), guardrail/injection blocking, context-window boundary.
  • PostgreSQL → token metering accuracy, spend attribution correctness, key/budget enforcement.
  • PrivateLink / region → cross-region egress blocked; packets stay in-region.
  • Bedrock/Claude → ZDR configuration verified; model-alias re-pointing with zero downtime.

Where to Go Next