Last week, several thousand AI engineers packed into San Francisco for the AI Engineer World's Fair. You'd expect the hallway conversation to be about the newest models, the flashiest agent demos, the next protocol. Some of it was. But the theme that kept surfacing — across talks, workshops, and booths — was quieter and more foundational than any launch: evals.
Not evals as an academic benchmark. Evals as the thing standing between a demo that works on stage and a system you can actually put in front of customers. Braintrust ran an observability workshop. IBM hosted a deep dive. Weights & Biases walked through building an end-to-end agent evaluation pipeline. The consensus in the room was almost unanimous: evaluation is the number one thing AI product builders have to get right, and it's still far too manual and far too hard.
That's the takeaway I want to unpack here, because it changes how you should architect an agentic system.
Evals are the alerting layer for software that doesn't fail loudly
Every engineer who has run a production service knows the discipline. When a microservice goes down, you find out immediately — a page fires, a dashboard turns red, an on-call engineer wakes up. Monitoring and alerting aren't optional; they're the connective tissue that lets you operate deterministic software at scale. You can't run production on hope.
Agentic software needs the same connective tissue — but the failure modes are different, and that's what makes this hard. A traditional service fails loudly: it returns a 500, it times out, the process crashes. An agent fails quietly. It returns a confident, well-formatted, completely wrong answer. It takes eight seconds when the user expected two. It subtly frustrates a customer who never files a ticket and just churns. Nondeterministic software doesn't throw an exception when it drifts — so you have to build the instrument that notices.
That instrument is evals. They are the alerting and monitoring layer for software whose failures are subjective rather than binary. And "subjective" is the whole challenge: there's no HTTP status code for is it hallucinating?, is the latency eroding trust?, is the user quietly giving up? You have to define what "healthy" means for your system, and then measure it continuously.
What an evaluator actually looks at
It helps to get concrete, because "evals" gets used as a vague catch-all. An evaluator is just a check that scores some dimension of your agent's behavior. A few of the most useful ones in production:
- Faithfulness / hallucination. For any system that answers from retrieved context, the strongest evaluators decompose the response into individual atomic claims, verify each claim against the source material, and score the proportion that are actually supported. A confident sentence with no grounding gets flagged before a customer ever reads it.
- Latency and cost per interaction. Speed is quality in an interactive product. An eval that tracks the latency distribution — not just the average, but the tail — tells you when your agent has quietly become too slow to trust.
- User frustration. Some of the most valuable signals are implicit: repeated rephrasings, abandoned sessions, escalations to a human, sentiment souring over a multi-turn conversation. An evaluator watching for these catches dissatisfaction that never becomes a support ticket.
- Task completion and tone. Did the agent actually do the thing? Did it stay on-brand and in-policy while doing it? These are the checks that matter most the moment an agent can take actions instead of just answering questions.
None of these are pass/fail in the way a unit test is. They're graded, contextual, and continuous — which is exactly why they map onto monitoring and alerting rather than onto CI's red-green checkmark.
LLM-as-judge: cheap, broad coverage
The breakthrough that makes this tractable at scale is the LLM-as-judge paradigm: use a separate, capable model to grade your agent's output against a clear rubric. Instead of paying a human to read every transcript, you write down what "good" looks like and let a judge model score thousands of interactions for cents each.
It works better than skeptics expect — on many tasks, LLM judges show strong agreement with human raters, and they scale to a volume no human review process ever could. For getting broad, cheap coverage across your system, it's the right first move.
But it has real failure modes, and pretending otherwise is how teams get burned. Rubrics that ask the judge to make subjective calls without concrete anchoring produce noisy, unreliable scores. And judge drift — where the judge's behavior silently shifts as your prompts, retrieval, or upstream models change — can rot your evaluation quietly, which is a particularly cruel irony for a system whose entire job is to catch quiet rot. A judge you never audit is just a second unmonitored agent.
Why you still need humans in the loop
Here's the part the demos gloss over: as your agentic system grows more complex, you cannot get to robustness on LLM judges alone. You have to pull in human annotators and labelers.
This isn't a failure of automation — it's the source of ground truth that keeps automation honest. Human domain experts are the ones who define what "correct" even means for your business, who label the hard edge cases, and who curate the datasets your judges are measured against. As one theme at the Fair put it bluntly: automated metrics cannot catch a model that is confidently lying in your brand's perfect tone of voice. A human who knows the domain can.
The mature pattern is a layered one. LLM judges give you cheap breadth. Human annotators give you expensive, high-trust depth on the cases that matter most — and their labels become the calibration set that tells you whether your judges can be trusted. The two aren't alternatives; they're the two ends of the same feedback loop.
The market is already consolidating around this
If you want a read on how central evals have become, watch where the capital and the acquisitions are going. This isn't a fringe tooling category anymore — it's becoming core infrastructure.
- Langfuse, the open-source platform for LLM observability, evaluations, and prompt management, was acquired by ClickHouse earlier this year, alongside ClickHouse's $400M Series D that pushed its valuation to roughly $15B. A database company buying an evals-and-observability company is a bet on owning the AI feedback loop.
- Arize ships Phoenix, an open-source, OpenTelemetry-based platform for tracing and evaluating agent runs with span-level visibility — the forensics layer for when something goes wrong deep in a multi-step agent.
- Braintrust takes an eval-first posture: it turns production traces into test cases and runs evals in CI/CD, blocking a deploy when quality drops so regressions get caught before users see them.
Three different bets — open-source observability, deep tracing, CI-gated evals — but all circling the same conviction the whole conference shared: evaluation is the control surface for agentic software.
Evals are also a cost discipline
There's a dimension of this that gets underrated, and it's the one I'd push hardest on with any team: evals aren't only about is it good? They're also about is it efficient?
Every token has a price, and an agentic system that reaches for a frontier model on every call is quietly burning money. You don't want to spend a frontier Fable token when a lightweight open-source model would have done the job just as well. But "just as well" is a claim — and the only honest way to make it is to measure it.
This is where evals earn their keep twice over. Run the same eval suite across different models and you frequently discover that a cheaper, faster model performs as well as the expensive one on your actual workload — sometimes better, once you account for latency budget, because a snappier response inside your users' patience window beats a marginally smarter one that arrives too late. Published routing work has shown roughly 2× cost reductions while holding around 95% of frontier quality, and teams that route intelligently commonly see 40–70% savings.
The catch is the same catch as everywhere else in this post: routing without measurement saves money and loses customers. The eval suite is what lets you cut cost safely — it's the gate that tells you quality held on the cheaper path before you ship it.
The feedback loop is the architecture
Step back and the shape of it is clear. There is a functionally unlimited universe of evals you could run against an agentic system — every dimension, every edge case, every model variant, every route. You cannot run all of them, and chasing coverage for its own sake is its own failure mode. The evals you choose have to be intentional. They have to encode what actually matters for your users and your business.
Done right, your evals aren't a QA afterthought bolted on before launch. They are the architecture of the feedback loop between your users and your system — the mechanism by which real-world behavior flows back in and makes the thing measurably better over time. That loop is what separates a team that knows its agents improved from a team that's guessing.
Pretty much every company in the world is building an agentic system in some form today. Which means pretty much every company needs to understand this — and most need someone on the team who understands it deeply, because the difference between an eval strategy that's intentional and one that's incidental is the difference between a system that compounds and one that merely works for a while.
AgentGraph designs agentic systems and places forward-deployed engineers with enterprise teams building exactly these feedback loops — the eval architecture that takes a system from promising demo to dependable production. If you're figuring out how to measure and improve your agents, let's talk.