LLM Tracing vs Evals: Which One Tells You Why Your Agent Broke

A user tells you the assistant gave them a wrong answer yesterday afternoon. You open your logs. You have the final response, a timestamp, and a request id. You do not have the documents that were retrieved. You do not have the tool the model chose, or the arguments it passed, or the instructions that were in front of it. You cannot reproduce it, because the retrieval index has been rebuilt twice since then.

That is the whole problem. The answer was wrong at one specific step, and you kept none of the steps.

The verdict

Tracing records what actually happened inside one request, step by step. Evals score output quality against expectations, across many requests. They are not competing tools and they do not overlap. If you can only build one this quarter, build tracing, because an eval score without a trace is a number you cannot act on. Knowing that faithfulness dropped from good to bad tells you to go and look. Tracing is the looking.

The reverse is survivable. Traces without evals leave you debugging one complaint at a time, which is slow and unpleasant, but it works. Evals without traces leave you holding a dashboard that says something got worse and no way to find out what.

What a trace actually is

A span is a record of one operation with a start time, an end time, and a bag of labelled values. A trace is a tree of spans covering one request end to end. This is not new. It is the same distributed tracing that has existed since services started calling other services. Treating it as an AI invention is how teams buy a second observability stack they did not need.

What is new is the vocabulary for the operations. OpenTelemetry keeps its generative AI conventions in a dedicated repository, open-telemetry/semantic-conventions-genai, split out from the core semantic conventions. It defines span shapes for inference, embeddings, retrievals, memory, and tool execution, and a set of attribute names under gen_ai.*.

The ones worth knowing on sight are gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.conversation.id for tying a thread together, and gen_ai.prompt.name with gen_ai.prompt.version for the template that produced the call. Span names follow {gen_ai.operation.name} {gen_ai.request.model}, so chat gpt-4 rather than something you invented.

Two things about the status of all this matter more than the attribute list. Every gen_ai.* attribute above is marked Development, which is OpenTelemetry's way of saying the names can still change. Only the borrowed core attributes such as error.type and server.port are Stable. So write the emitting code behind a thin function of your own rather than sprinkling attribute strings through your handlers, and a rename costs you one file.

The second thing is that this is a wire format, not a product. Emitting spans that follow the convention keeps you able to switch backends later, which is the only real defence against picking the wrong vendor early.

What an eval actually is

An eval is a scoring function applied to a model's output. It takes the input, the output, sometimes the retrieved context, and returns a number or a label. That is the entire idea, and the reason it feels harder than it is comes from the scoring, not the plumbing.

Three kinds are worth separating, because they have very different costs. A deterministic check is code: did the output parse as JSON, did it contain a required field, did it stay under a length. A reference check compares against a known correct answer, which requires you to have one. A model-graded check asks another model whether the output was faithful to the retrieved context, and it is the expensive one, because every score is an inference call.

The trap is starting with model-graded evals because they are the interesting ones. Start with the deterministic checks. They are nearly free, they never drift, and in most systems they catch more real defects than anyone expects. A model that returns malformed JSON one time in fifty is a much more common production problem than a model that is subtly unfaithful.

An eval is also a claim about your data, not about your model. If your dataset is thirty examples you wrote in an afternoon, a score moving from twenty-six to twenty-eight is noise. Treat the dataset like a test suite: it grows from real failures, and the best entries are the ones a user complained about. The same discipline that makes SLOs, SLAs and SLIs useful applies here. A measurement nobody agreed the meaning of in advance becomes a debate instead of a decision.

Pick tracing when

You cannot reproduce a complaint. You have more than one model call per request, which means anything using tools or retrieval. You want to know where the latency is, and you suspect it is not the model. You are running any kind of agent loop, where the interesting failure is a decision made three steps in.

That last one is the real case. In a single-call system you can usually reconstruct what happened from the request and response. The moment there is a loop, the state that caused the failure is intermediate. It is discarded on the way out, and logging the final answer never recovers it. The shape of that loop decides how legible the trace will be. I have made the same point about agentic systems and about multi-agent topologies.

Pick evals when

You are about to change a prompt, a model, or a retrieval parameter, and you need to know whether it made things worse. That is the whole case, and it is a good one. Prompt changes are the least reviewable edits in modern software. The diff is prose, the effect is statistical, and nobody can tell by reading it whether it is better.

Evals are how a prompt change becomes reviewable. Without them you are merging on vibes, and every engineer on the team knows it and says nothing.

The dangerous part: what goes into the span

Read this section twice.

The most useful thing to put on an inference span is the thing you must be most careful with. OpenTelemetry's own guidance is explicit: model instructions, user messages and model outputs are treated as sensitive and large, and instrumentations should not capture them by default, only offer an opt-in. The attributes exist, gen_ai.system_instructions, gen_ai.input.messages and gen_ai.output.messages, and the specification names three patterns in increasing order of maturity. Record nothing. Record content on the span. Store content externally and put a reference on the span.

The third one is what production wants, and almost nobody starts there. The reason given in the specification is exactly right: external storage lets you apply a different access model to user content than to operational telemetry. That distinction is the point. Your trace store is a debugging tool, so half the engineering team can usually read it. Your users' messages are not a debugging tool, and the same half of the team should not be able to read those by default.

So the failure is not a leak in the dramatic sense. Nothing throws, nothing alerts. You turn on content capture in staging because it is genuinely useful. You promote the config. Six months later user conversations are sitting in a system scoped for latency graphs, with a retention policy nobody chose and an audit story nobody can tell.

The size problem is the same decision arriving from a different direction. Message content is frequently larger than backend limits for a single attribute value, and it may contain media. The specification allows truncation that preserves the JSON structure. Turn it on. A span silently dropped for being oversized is worse than one carrying half the content. You lose the record of the request you most wanted, since the biggest requests are usually the interesting ones.

The rule I apply: prompts and outputs never land on a production span without a named retention period and a named access control. Both get written down before the flag goes on.

Knowing whether it works

Both systems have a way of looking healthy while telling you nothing.

For tracing, the number to watch is not span volume, it is whether a span tree is complete. A trace missing its retrieval spans is not a trace, it is a receipt. Pick a real complaint, open its trace, and ask whether you could explain the answer to the user from what is on screen. If not, the instrumentation is decorative.

Sampling is where this usually breaks. Head sampling, deciding at the start of the request, will throw away exactly the slow and failing requests you needed, because it decides before it knows. Prefer keeping everything that errored or ran long, and sampling the boring successes.

For evals, watch how often the score changes when nothing changed. Run the same dataset twice against an unchanged system. A model-graded eval that returns a different number each time is measuring its own judge, and you will spend a week chasing a regression that was never there.

Cost is the other honest number, and it is easy to miss because it hides inside the same bill as your product traffic. A model-graded eval on every request doubles your inference calls. That is a real decision about token spend, not an afterthought, and the usual answer is to run the expensive evals on a sample and the free ones on everything.

Where this stops working

Neither of these tells you what a good answer is. A trace shows you the model was given three documents and cited the second one. An eval tells you a judge scored it four out of five. Whether that answer helped the person who asked is a question neither system can reach, and on anything with real stakes, retrieval quality is the part that decides it. A retrieval cutoff that is too loose returns a plausible document about the wrong subject. The model uses it faithfully. The faithfulness eval passes, and the user gets a confident wrong answer with a citation attached. Every layer reports success.

That is the case that keeps me honest about tooling. Observability tells you what your system did and how consistently it did it. It does not tell you that the thing you built is the right thing.

Build tracing first, keep prompts out of your spans until someone owns the retention policy, and treat any eval score you cannot open a trace from as decoration.