At a glance
Hamel Husain opens with a claim he has repeated in writing many times since: the number one reason AI products fail is the absence of an evaluation system, and the teams that ship successfully are not the ones with the best prompts, they are the ones with the fastest loop between noticing a failure and confirming they fixed it. He is joined by Emil Sedgh, CTO at Rechat, which gives the talk something most eval talks lack: a real product, Lucy, an assistant embedded in a real estate CRM, with real users and real failures.
The structure they lay out has three levels: cheap assertions that run in continuous integration, human review paired with an LLM judge that has been aligned against that human review, and production A/B tests with real user feedback. Underneath all three sits the unglamorous centerpiece, a custom built tool for looking at your own data, because the thing that actually stops teams from improving is friction in looking at data.
The talk is aimed squarely at people who have shipped a demo and cannot tell whether their changes are making it better.
The deep explanation
The product, and why generic evals did not work
Lucy lives inside a real estate CRM and handles a wide surface: finding contacts, searching listings, creating email campaigns, scheduling. That breadth is exactly what makes generic benchmarks useless. Nobody publishes a leaderboard for "did the assistant correctly build a marketing email for a listing under a client's brand." The evaluation has to be constructed for the domain, which is the talk's title and its thesis.
Their first attempt looked like everyone's first attempt: try things, eyeball the output, ship, get complaints, tweak the prompt, repeat. The failure mode of that loop is that it has no memory. A prompt change that fixes one case silently breaks three others, and nobody finds out until a user does.
Level one: assertions and unit tests
The first level is deliberately unsophisticated. Write code that checks properties of an output that must always hold, and run it in CI on every change.
For Lucy that means things like: if the assistant was asked to write an email, does the output contain valid markup that the CRM can render, does it reference a listing that actually exists, are placeholder variables filled in, is the required disclaimer present, did it avoid inventing a phone number. None of these require a model to evaluate. They are string checks, schema validations, and lookups.
The arguments the speakers make for taking this level seriously:
- It is fast and free, so it can run on every commit and catch regressions in seconds.
- The failures it catches are real. A surprising fraction of production problems in LLM products are structural, not semantic.
- Writing assertions forces specificity. You cannot write the assertion until you have decided what correct means, and many teams have never done that.
- The assertions come from looking at failures. Every time you find a bad output, you turn it into an assertion so it can never come back silently.
Level two: human review and an aligned LLM judge
Assertions cannot tell you whether an email was any good. That needs judgment, and judgment starts with humans.
The step everyone skips is looking at the data. Husain is emphatic here, and this is the part of the talk with the most practical payoff. Rechat built a purpose made viewer for their own traces: the input, the tool calls, the intermediate reasoning, the final output, all on one screen, with a place to record a verdict and a note. Not a dashboard of aggregate metrics, and not a general purpose observability tool with fifteen clicks between you and a trace. Removing friction is the whole design goal, because a review tool that is even slightly annoying does not get used, and the review does not happen.
Once humans have labeled a set of traces, you can build the LLM judge. The critical discipline, and the thing that separates a useful judge from a fake one, is alignment: the judge is not trusted because it is a big model, it is trusted only to the extent that its verdicts agree with the human verdicts on the labeled set. You measure that agreement. If it is poor you rewrite the judge prompt, usually by giving it the specific criteria and few shot examples drawn from the human labels, and measure again. When agreement is high enough, the judge can be run over far more traces than a human could review, and it inherits credibility from the alignment measurement rather than from vibes.
Two supporting practices come with it. Keep re measuring alignment on fresh human labels, because both the product and the failure distribution move. And be honest that the judge is a proxy: a high judge score with no human spot checking is a metric that will eventually be gamed by your own prompt iteration.
Synthetic data, and the coverage problem
A related problem: you cannot wait for real users to produce the test cases you need, especially before launch or for a rarely used feature. So they generate them.
The method is systematic rather than freeform. Enumerate the dimensions of the product surface, the features, the entity types, the user personas, the scenario shapes, then generate combinations and turn each into a realistic user query with a model. That gives broad, structured coverage instead of the handful of examples a developer would think of, and it surfaces the interactions between features that are exactly where the bugs live.
The generated queries are run through the system and the outputs enter the same review pipeline as real traffic.
Level three: production
The last level is the only one that settles anything. Ship the change to a fraction of traffic, measure what users actually do, and collect explicit feedback where you can. The speakers are realistic about it: production signal is slow, noisy, confounded, and expensive to obtain, which is exactly why the first two levels exist. You use levels one and two to arrive at production with changes you already believe in, and you use production to check whether your proxies were telling the truth.
The data flywheel, and the payoff beyond quality
The closing argument widens the frame. Once you have this pipeline, you are not just measuring, you are producing curated data as a byproduct: labeled failures, verified good outputs, structured traces. That is the raw material for fine tuning a smaller and cheaper model to do what your prompted frontier model does, and it is why they describe evaluation as an unlock rather than an overhead. Teams without evals cannot fine tune, because they have no trustworthy data and no way to tell whether the fine tune helped.
The other repeated theme is tooling minimalism. Most teams do not need a platform, they need to remove the friction between themselves and their own data, and the fastest route to that is usually a small purpose built viewer written in an afternoon. Buying an observability product before you have looked at fifty traces by hand is a way of avoiding the work rather than doing it.
Key takeaways
- The most common root cause of a failed AI product is the absence of an evaluation system, not a weak model or a bad prompt.
- Level one is assertions: cheap deterministic checks that run in CI. A large share of real failures are structural and get caught here.
- Level two is human review plus an LLM judge, and the judge is only worth anything to the extent its verdicts have been measured against human verdicts on labeled data.
- Build your own trace viewer. Friction in looking at data is what actually stops teams from improving, and a purpose built tool removes it.
- Generate synthetic test cases systematically across features, entities, and personas rather than relying on the handful of examples a developer would think of.
- Level three is production A/B and user feedback: the only real verdict, and too slow to iterate against, which is what the first two levels are for.
- Every failure found by eye should become a permanent automated check. That ratchet is the entire value of the system.
- Evals produce curated data as a byproduct, which is what makes fine tuning a cheaper model possible later.
Where this sits in the LLM Learning track
The previous video ends with a warning about believing your system works because you tested it on examples you chose. This is the answer to that warning, and it is the least glamorous and most load bearing hour in the track. Read it before the agent building video, because an agent multiplies the number of places a system can go wrong and is close to unmaintainable without the loop described here.
Resources mentioned
- Hamel Husain's blog, especially Your AI Product Needs Evals and Creating a LLM-as-a-Judge That Drives Business Results
- Rechat, the real estate platform behind the Lucy assistant
- AI Engineer, the conference where the talk was given
- Evals course by Hamel Husain and Shreya Shankar
- Parlance Labs, Husain's consultancy, which publishes further material on this pipeline
About this page
This reconstruction was built from the public record of the talk and the speakers' published writing on the same system rather than from its caption track, because the machine that assembled it had no network route to YouTube. The three levels, the flywheel, and the practices are here; the verbatim transcript, the timestamped chapter map, and pull quotes are not, and those get backfilled on the next run from a machine that can reach the source. Until then, watch it on YouTube alongside this page.
