LLM Observability: What to Log and Trace

Quick answer
LLM observability is the ability to reconstruct what a request sent, which model handled it, what it returned, what it cost, and whether the outcome was good.
A trace per request needs: prompt template version, model and parameters, token counts, tool calls, latency per step, cost, and a success signal.
Log identifiers and metadata by default, and full prompt and response content only under a redaction and retention policy.
Standard spans and attributes matter: OpenTelemetry’s GenAI conventions let the same trace correlate with the rest of your services.
Observability without an outcome signal tells you the call succeeded, not that the feature worked; capture user feedback and business success alongside the trace.

What should an LLM trace contain?

One trace per user-visible request, with child spans for retrieval, each model call, tool execution, and post-processing. Each model span should carry the provider, model, prompt template version, token counts for input and output, cached-token counts, latency, and estimated cost. Tool spans should carry the tool name, arguments summary, duration, and whether it succeeded.

The trace should also carry the outcome: was the answer accepted, did the user regenerate, did validation fail, did the request escalate to a stronger model. Without that field, you can measure performance but not quality, and quality is what users experience.

How do you handle prompts and responses safely?

Treat prompt and response content as sensitive by default. Either do not store it, or store it under a policy with encryption, access control, short retention, and redaction of personal data. A request log that contains full customer conversations is a liability that grows with every request.

In practice, store content only for sampled requests or for failures, and store structure for all of them. A hash of the prompt, the template version, the token counts, and the tool names are usually enough to debug a latency or cost problem without retaining what the user said.

# One span per model call, with attributes that answer cost and quality questions.
with tracer.start_as_current_span("llm.call") as span:
    span.set_attribute("gen_ai.system", "anthropic")
    span.set_attribute("gen_ai.request.model", model)
    span.set_attribute("gen_ai.usage.input_tokens", usage.input_tokens)
    span.set_attribute("gen_ai.usage.output_tokens", usage.output_tokens)
    span.set_attribute("app.prompt_version", prompt_version)
    span.set_attribute("app.cost_usd", cost)
    span.set_attribute("app.outcome", outcome)   # accepted | retried | failed

What should you alert on?

Alert on user-visible symptoms: error rate, timeout rate, p95 latency, and the share of requests that fail validation or get regenerated. Then add cost anomalies: spend per successful task above a threshold, a sudden rise in input tokens, or a drop in cache hit rate.

Do not alert on token counts alone. A spike in tokens may be a legitimate long document or a broken prompt; the surrounding trace tells you which. Alert on the outcome and investigate the cause in the trace, rather than paging on a proxy.

How do you close the quality loop?

Capture feedback where the user already gives it: accepted or edited, thumbs up or down, regenerated, escalated to a human. Join that signal to the trace and to the prompt version. Then a change in quality can be attributed to a specific prompt, model, or route instead of to a vague sense that the feature got worse.

Maintain a small evaluation set from real failures and run it before every prompt or model change. The trace tells you what happened; the eval tells you whether the next version is better. Neither replaces the other, and teams that only have dashboards tend to change prompts without evidence.

Where this bites vibecoders
Generated AI code logs the prompt only when something breaks, which means the logs contain some full prompts and no structured data. Adding a span with model, token counts, prompt version, cost, and outcome is a small change that makes cost and quality measurable without keeping every conversation.

Where AI coding assistants get this wrong

Logging full prompts and responses indefinitely with no redaction.
Recording latency but not tokens, cost, or the model version.
Using a free-form log line instead of structured attributes.
Omitting the outcome, so quality cannot be joined to the trace.
Alerting on token count spikes without a user-visible symptom.
Changing prompts without an evaluation set or a version identifier.

Checklist

Define one trace per user-visible request.
Record model, prompt version, tokens, latency, cost, and tools per call.
Record the outcome and user feedback.
Redact or avoid storing sensitive content by default.
Alert on error rate, latency, validation failures, and cost per success.
Keep an evaluation set built from real failures.

FAQ

Do I need a dedicated LLM observability vendor?

Not necessarily. If you already run OpenTelemetry, the GenAI semantic conventions let you attach model spans to your existing traces and dashboards. A dedicated tool helps with prompt-level analysis and evaluation, but the first step is structured tracing, which you can do with what you have.

Is it safe to log prompts?

Only under a policy. Prompts can contain personal data, credentials pasted by users, and proprietary documents. Store them encrypted with access control and short retention, or store a hash and the structure instead. The debugging value rarely justifies retaining every full prompt forever.

What is the most useful metric?

Cost per successful task, paired with the validation or acceptance rate that defines success. It combines spend and quality in one number and moves when either changes. Latency and error rate are necessary but do not tell you whether the answer was usable.

How do traces help with prompt changes?

They let you compare the same feature before and after by prompt version: tokens, latency, cost, validation failures, and acceptance. Without versioning, a change in behaviour is indistinguishable from a change in traffic. Version the prompt like code and attach the version to every span.