AI Engineer
How to Construct Domain Specific LLM Evaluation Systems: Hamel Husain and Emil Sedgh
Hamel Husain and Emil Sedgh argue that most failed AI products share one root cause, no evaluation system, and then show what a real one looks like using Lucy, the assistant Rechat shipped into a real estate CRM. Three levels: cheap assertions that run in CI, human review and an LLM judge that has been aligned against that human review, and production A/B with real user feedback. The unglamorous centerpiece is a custom tool for looking at your own data, because friction in looking at data is what actually stops teams from improving.