LLM Observability — LiteLLM & Langfuse

What this is: a concepts-and-tutorial guide to the two tools most teams pair for LLM observability — LiteLLM (the gateway that routes and meters every model call) and Langfuse (the platform that traces, logs, and lets you see the problems). Covers how observability works, how usage/cost is tracked across every model, how logs are captured, and how you debug what went wrong.

Fits after: AI Test Strategy and the evaluation guides — observability is how you test the system in production, not just in CI.

---

1. Why LLM Observability Is a Distinct Problem

A traditional web service is observable with request logs, metrics, and traces. LLM apps add failure modes those tools can't see:

LLM-specific concern	Why standard APM misses it
Cost per call varies	Billed per token, not per request — a "200 OK" can cost $0.001 or $2.00
Non-determinism	Same input → different output; you must log the actual prompt+response to reproduce
Multi-provider sprawl	Calls fan out to OpenAI, Anthropic, Bedrock, local models — no single usage view
Quality is invisible in status codes	A hallucinated answer is a 200 — you need the payload + an eval score to know it failed
Nested pipelines	One user request = retrieve + rerank + N LLM calls (agents) — you need a trace tree, not flat logs
Prompt drift	A prompt template change silently shifts quality/cost — needs versioned prompt tracking

The two-tool answer: put a gateway in front of every model call to standardize and meter it (LiteLLM), and pipe every call into an observability platform that stores the full trace and surfaces problems (Langfuse).

flowchart LR
    APP["🧩 Your app / agent"] --> LL["🚪 LiteLLM gateway<br/>unify · route · meter · budget"]
    LL --> P1["OpenAI"]
    LL --> P2["Anthropic"]
    LL --> P3["Bedrock"]
    LL --> P4["local / vLLM"]
    LL -. "callback: every request+response,<br/>tokens, cost, latency, errors" .-> LF["📊 Langfuse<br/>traces · logs · usage · dashboards · alerts"]

---

2. LiteLLM — The Unified LLM Gateway

2.1 Concept

LiteLLM gives you one OpenAI-compatible interface to 100+ LLM providers. Instead of learning each vendor's SDK, you call one API and swap models with a string. It ships two ways:

• Python SDK (litellm) — a drop-in completion() function you call from code.
• Proxy server (LiteLLM Gateway) — a standalone service every app points at; centralizes keys, budgets, routing, logging, and cost tracking for a whole org.

Capability	What it does
Unified API	model="gpt-4o" or model="claude-sonnet" — same call shape, OpenAI format
Routing & load balancing	Spread load across deployments/regions; pick by latency or cost
Fallbacks	Primary model errors/rate-limits → auto-retry on a backup model
Virtual keys	Issue per-team/per-user keys with their own limits — without sharing provider keys
Budgets & rate limits	Hard spend caps and RPM/TPM limits per key/user/team
Cost tracking	Computes $ cost per call from token counts; writes to a Postgres DB
Logging callbacks	Forwards every call to 20+ observability backends (Langfuse, OTEL, Datadog, S3…)

2.2 Tutorial — SDK quickstart

# pip install litellm
from litellm import completion
import os
os.environ["OPENAI_API_KEY"]    = "..."
os.environ["ANTHROPIC_API_KEY"] = "..."

# same call, different providers — just change the model string
r1 = completion(model="gpt-4o",          messages=[{"role":"user","content":"Hi"}])
r2 = completion(model="claude-sonnet",   messages=[{"role":"user","content":"Hi"}])

print(r1.choices[0].message.content)
print(r1.usage)            # prompt_tokens / completion_tokens / total_tokens
print(r1._hidden_params["response_cost"])   # computed $ cost for THIS call

2.3 Tutorial — the Proxy Gateway (org-wide control plane)

A config.yaml defines model aliases, then you run the proxy; apps point their OpenAI client at it:

# config.yaml
model_list:
  - model_name: gpt-4o                       # the alias apps call
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude                       # alias → different provider
    litellm_params:
      model: anthropic/claude-sonnet-4
      api_key: os.environ/ANTHROPIC_API_KEY

litellm_settings:
  success_callback: ["langfuse"]             # ← log every OK call to Langfuse
  failure_callback: ["langfuse"]             # ← AND every error

router_settings:
  routing_strategy: latency-based-routing    # pick fastest healthy deployment
  fallbacks: [{"gpt-4o": ["claude"]}]        # gpt-4o fails → retry on claude

litellm --config config.yaml       # starts the gateway on :4000

# apps use the standard OpenAI SDK, pointed at the proxy + a VIRTUAL key
from openai import OpenAI
client = OpenAI(base_url="http://localhost:4000", api_key="sk-team-frontend-key")
client.chat.completions.create(model="gpt-4o", messages=[{"role":"user","content":"Hi"}])

Now every call in the org is routed, budgeted, cost-tracked, and — via the callbacks — logged to Langfuse, with no app code change.

---

3. Langfuse — The Observability Platform

3.1 Concept — the trace data model

Langfuse is an open-source LLM observability & tracing platform. Its power is a hierarchical trace model that mirrors how LLM apps actually run:

TRACE  (one user interaction, e.g. "answer this support ticket")
 ├── SPAN: retrieve_context          (a step; has latency)
 │    └── GENERATION: embed_query     (an LLM/embedding call: input, output, tokens, cost, model)
 ├── SPAN: rerank
 └── GENERATION: final_answer         (the LLM call: prompt, completion, tokens, $, latency)
        ├── attached: SCORE (faithfulness=0.9)   ← eval result
        └── attached: metadata (userId, sessionId, tags, prompt version)

Object	Represents
Trace	One end-to-end request/interaction — the root you search and inspect
Span	A non-LLM step (retrieval, tool call, business logic) with timing
Generation	An LLM/embedding call — captures input, output, model, token counts, cost, latency
Score	A quality metric attached to a trace/generation (eval result, user 👍/👎)
Session	Groups multiple traces into a conversation
User	Attribution for per-user usage/cost

3.2 Tutorial — instrument code directly

Three common integration styles:

# (a) decorator — auto-creates a trace/span around any function
from langfuse.decorators import observe
from langfuse.openai import openai       # drop-in OpenAI wrapper: auto-logs generations

@observe()
def answer(question: str):
    ctx = retrieve(question)             # becomes a span
    resp = openai.chat.completions.create(   # becomes a generation (tokens+cost auto-captured)
        model="gpt-4o",
        messages=[{"role":"user","content": f"{ctx}\n\n{question}"}],
    )
    return resp.choices[0].message.content

# (b) attach usage attribution + a quality score to the current trace
from langfuse.decorators import langfuse_context
langfuse_context.update_current_trace(user_id="u_123", session_id="s_9",
                                      tags=["support", "prod"])
langfuse_context.score_current_trace(name="faithfulness", value=0.92)

Langfuse also plugs into LangChain (callback handler), LlamaIndex, and — most relevantly here — LiteLLM (§4).

---

4. The Integration — LiteLLM → Langfuse (the core of this guide)

This is how you "track usage in the right model, capture the logs, and see the problems" with zero per-call code.

4.1 Wiring it up

Set the callback (as in §2.3) plus three env vars, and every call LiteLLM handles is logged:

export LANGFUSE_PUBLIC_KEY="pk-lf-..."
export LANGFUSE_SECRET_KEY="sk-lf-..."
export LANGFUSE_HOST="https://cloud.langfuse.com"   # or your self-hosted URL

# SDK style — same effect without the proxy
import litellm
litellm.success_callback = ["langfuse"]
litellm.failure_callback = ["langfuse"]     # errors are logged too — critical for debugging

Two integration transports exist: the standard callback (default, simplest) and a newer OpenTelemetry (OTEL) transport if you want to fan out to multiple OTEL backends. Both capture the same data.

4.2 What gets captured on every call

flowchart LR
    REQ["LLM call via LiteLLM"] --> CAP["Captured automatically"]
    CAP --> M["model + provider actually used<br/>(after routing/fallback)"]
    CAP --> T["prompt + completion (full text)"]
    CAP --> U["token counts (in / out / total)"]
    CAP --> C["computed $ cost"]
    CAP --> L["latency (start→end)"]
    CAP --> E["errors / exceptions / which fallback fired"]
    M & T & U & C & L & E --> LF["Langfuse trace"]

4.3 Grouping calls into meaningful traces & usage buckets

Pass Langfuse metadata through LiteLLM so calls are attributed correctly — this is how you track usage "in the right model" and per user/team:

completion(
    model="gpt-4o",
    messages=[...],
    metadata={
        "generation_name": "support-answer",
        "trace_id": "ticket-4821",     # group the retrieve+rerank+answer calls into ONE trace
        "session_id": "conv-91",
        "trace_user_id": "user_123",   # → per-user cost/usage dashboards
        "tags": ["prod", "support"],
    },
)

Langfuse then directly captures the cost information LiteLLM returns — so token & cost tracking is exact per model, per key, per user, per team, and per session.

---

5. How Observability Actually Works — The Three Jobs

5.1 Track usage across every model

Because LiteLLM meters each call and Langfuse aggregates it, you get dashboards answering:

• Which models are being used, and how often (after routing/fallback — you see the actual model, not just the requested one).
• Token & cost totals sliced by model / provider / user / team / tag / session / time.
• Budget burn — LiteLLM enforces hard caps per virtual key; Langfuse visualizes the trend so you see a spend spike before the cap is hit.

Key point on "the right LLM": with fallbacks enabled, a request to gpt-4o may actually be served by claude. The trace records the model that truly answered, so usage/cost is attributed to the real provider — not the one you asked for. This is impossible to get right by logging in app code before the gateway decides.

5.2 Capture the logs

Every request+response is stored as a generation inside a trace: full prompt, full completion, model, tokens, cost, latency, and any error. Because it's the actual routed call, the log is reproducible — you can copy the exact prompt from a failed production trace into a playground and replay it. Nested pipelines (RAG, agents) appear as a trace tree, so you see the retrieval step, each tool call, and each LLM generation in one timeline rather than scattered log lines.

5.3 See the problems

This is where observability earns its keep. In Langfuse you find:

Problem	How it surfaces
Errors / failures	failure_callback logs exceptions, timeouts, rate-limits — filter traces by error status
Latency outliers	Sort generations by latency; find the slow model/step; inspect TTFT
Cost spikes	A user/prompt burning tokens shows up in cost-by-dimension charts
Quality regressions	Attach eval scores (faithfulness, relevance) to traces; dashboard the trend; a prompt change that drops scores is visible
Fallback storms	Repeated fallback firing = a provider is degraded; visible in model-actually-used stats
Bad prompts	Versioned prompts + scores show which prompt version regressed quality/cost
User-reported issues	Search the exact trace_id/session_id to pull the full conversation and replay it

flowchart LR
    PROD["🌐 production traffic"] --> LF["📊 Langfuse"]
    LF --> D1["🔴 filter: errors → fix crashes"]
    LF --> D2["🐢 sort: latency → fix slow steps"]
    LF --> D3["💸 group: cost → find token hogs"]
    LF --> D4["📉 scores: quality → catch regressions"]
    D1 & D2 & D3 & D4 --> ACT["🔧 reproduce trace → fix → verify score recovers"]

---

6. LiteLLM vs Langfuse — Complementary, Not Competing

	LiteLLM	Langfuse
Role	Gateway / control plane (the pipe)	Observability / analysis (the lens)
Sits	In the request path (every call goes through it)	Beside the path (receives copies of calls)
Core value	Unify providers, route, budget, meter, enforce	Trace, log, visualize, score, debug
Answers	"Call any model safely and within budget"	"What happened, what did it cost, why did it fail?"
Without the other	Metering but weak visualization/trace tree	Great traces but you instrument each provider yourself

Together: LiteLLM standardizes and meters at the choke point; Langfuse turns that stream into searchable traces, usage dashboards, and quality scores. One line of callback config connects them.

You don't strictly need both. Langfuse can instrument code directly (decorators/OpenAI wrapper) without LiteLLM; LiteLLM can log to other backends. But the pairing is the common production pattern: gateway for control, Langfuse for insight.

---

7. The QA / Testing Angle

Observability is production testing — and it feeds CI testing:

• Traces as test evidence — a failing production trace becomes a regression test case: copy its exact input into your golden dataset.
• Scores close the loop — run LLM-as-judge evals over sampled production traces and write the scores back to Langfuse; now quality is monitored continuously, not just at release.
• Datasets & experiments — Langfuse datasets let you replay a fixed input set against a new prompt/model and compare scores — the observability tool doubling as an offline eval harness.
• Cost as a test gate — assert that a prompt change didn't blow up token usage by comparing cost-per-trace before/after.

---

8. Quick Reference

Need	Where
Call 100+ models, one API	LiteLLM SDK completion()
Org-wide keys/budgets/routing	LiteLLM Proxy config.yaml
Log every call to Langfuse	success_callback + failure_callback = ["langfuse"]
Per-user/team cost	metadata: trace_user_id / tags + Langfuse dashboards
Group a RAG/agent pipeline	shared trace_id across calls (or @observe)
See errors	failure_callback + filter traces by status
Catch quality regressions	attach scores + trend dashboard
Reproduce a bug	open the trace → copy exact prompt → replay

---

Where to Go Next

• AI Test Strategy — where production observability fits the overall plan
• Cloud LLM Evaluation Tools — the eval scores you write back into traces
• RAG Evaluation Metrics — the metrics you monitor in production
• Enterprise LLM Platforms — the providers LiteLLM unifies
• Official docs: LiteLLM · Langfuse · LiteLLM ↔ Langfuse integration