This course teaches engineers and technical product managers how to evaluate AI applications through data-driven testing and systematic error analysis.
The course covers evaluation across the LLM application lifecycle, from development to production. Topics include instrumentation and observability, error analysis and failure categorization, synthetic data generation, annotation strategies, and defining metrics using code-based methods. It is delivered as a flipped classroom: lectures are recorded, with live office hours and a Discord community for interaction.
It suits engineers and technical PMs who are comfortable coding or vibe coding and want hands-on practice with exercises, code, and data.