LLM Tracing vs Evals: Which One Tells You Why Your Agent Broke

A user tells you the assistant gave a wrong answer. You check logs and find the final response, a timestamp, and a request id. You lack the retrieved documents, the tool calls, and the prompt state. The retrieval index changed, so you cannot reproduce the bug.

The answer failed at one specific step, and none of the intermediate steps were preserved.

The verdict

Tracing records what happened inside one request, step by step. Evals score output quality against expectations across many requests. They solve different problems. If you can only build one first, build tracing: an eval score without a trace is a metric you cannot debug.

Traces without evals require fixing issues one complaint at a time. Evals without traces leave you with a dashboard showing quality drops with no way to find the root cause.

What a trace actually is

A span is a timed record of a single operation with labelled metadata. A trace is a tree of spans covering an entire request. This is standard distributed tracing applied to AI systems.

OpenTelemetry maintains generative AI standards in open-telemetry/semantic-conventions-genai:

• gen_ai.operation.name: The operation type (chat, embeddings, execute_tool).
• gen_ai.provider.name and gen_ai.request.model: Upstream vendor and model identifiers.
• gen_ai.conversation.id: Thread identifier for multi-turn sessions.
• gen_ai.prompt.name and gen_ai.prompt.version: Template identifier and version.

Two rules keep instrumentation maintainable:

• Wrap gen_ai.* attributes in helper functions: they are marked Development and subject to upstream naming changes.
• Use open wire standards rather than proprietary vendor SDKs to avoid backend lock-in.

What an eval actually is

An eval is a scoring function applied to a model output. It evaluates inputs, outputs, and retrieved context to produce a score or label.

Three categories exist:

• Deterministic checks: Code assertions (JSON validation, required keys, regex patterns, character limits). Fast, free, and stable.
• Reference checks: Comparisons against golden datasets with known correct answers (exact match, BLEU, ROUGE, embeddings).
• Model-graded checks: Prompts sent to an LLM judge to evaluate qualitative properties (faithfulness, relevance, tone). Flexible, but add inference latency and cost.

Start with deterministic checks. They catch common production failures, like malformed JSON or missing arguments, without adding model costs.

Treat eval datasets like test suites: expand them with real user failures and edge cases. The discipline that governs SLOs, SLAs and SLIs applies here. Undefined metrics lead to debate rather than resolution.

Pick tracing when

• You cannot reproduce a user issue from final inputs and outputs alone.
• Requests make multiple LLM calls (routing, tool use, or multi-step retrieval).
• You need to profile latency across model calls, external APIs, and databases.
• You run agent loops where failures happen in early intermediate steps.

In multi-step systems, intermediate state is discarded before the final response is returned. Logging only the final completion loses the context needed for debugging, as detailed in agentic systems and multi-agent topologies.

Pick evals when

• You are modifying system prompts, templates, or model parameters.
• You are changing model versions or swapping providers.
• You are tuning retrieval components (chunk sizes, embeddings, rerankers).
• You need automated pull request checks in CI to catch quality regressions.

Prompt changes are difficult to assess through code review alone. Evals turn probabilistic behavior into measurable pass rates.

The dangerous part: what goes into the span

Model instructions, user messages, and completions contain sensitive data and should not be captured indiscriminately in trace backends.

Three payload architectures exist:

• Record nothing: Capture latencies, token counts, and status codes only. Safe, but limits root-cause debugging.
• Record payload on the span: Store full messages directly in span attributes. Simple, but exposes private user conversations to anyone with telemetry access.
• Store payloads externally: Write full messages to encrypted object storage with dedicated access controls and retention policies. Attach only reference IDs to telemetry spans.

Store payloads externally in production. Telemetry stores are accessible to engineering teams for debugging; user conversations require stricter access governance.

Enforce JSON-safe truncation on oversized inputs to prevent spans from being dropped silently by telemetry backends.

The rule: prompts and completions never enter production spans without explicit retention limits and defined access controls.

Knowing whether it works

Both systems can appear healthy while failing to provide actionable signal.

For tracing:

• Span completeness: Confirm traces capture the complete call graph, including vector lookups, tool calls, and model requests.
• Tail sampling: Retain all failed and slow requests, while sampling standard successes. Head sampling discards the errors you need to inspect.

For evals:

• Judge consistency: Run identical datasets twice on an unchanged system. High variance indicates an unstable judge prompt.
• Inference overhead: Model-graded checks on every request double token usage. Run expensive evals on scheduled benchmarks or sampled traffic to manage token spend.

Where this stops working

Neither tracing nor evals can verify whether an answer solved the user problem.

A trace shows the model cited the second retrieved document. An eval confirms the answer was faithful to that document. If retrieval returned documents for the wrong topic, the model generates a cited, fluent, incorrect response. Every metric reports success while the user receives bad data.

Observability verifies system execution and consistency. It does not replace product validation.

Build tracing first, keep raw user prompts out of open telemetry backends, and treat ungrounded eval scores as unverified metrics.