Automated Evaluation

What is automated evaluation?

Human evaluation means reading one sample at a time — slow and expensive. Automated evaluation puts a program or another model in the examiner's chair: it scores output, catches errors and judges quality automatically, churning through thousands of samples in seconds.

The common approaches

Rules and metrics
Exact match, BLEU, ROUGE and friends measure how close a machine's answer is to a reference — good for objective tasks.
A model as judge
Use a stronger LLM to score another model's answers. Called LLM-as-a-Judge, it can assess softer dimensions like tone and coherence.
Assertion-style tests
Set hard constraints on output — "must return valid JSON", "no profanity" — and fail it when violated.

The good and the bad

Upside: fast, cheap, scalable
Change a prompt and see full test results in seconds — something humans simply can't do.
Downside: it can miss the point
Machine scores don't always match human taste, especially on subjective stuff like creativity or humor.

How to use it well

The usual pattern is "automation for coverage, humans for judgment": run automated evaluation for fast feedback in daily iteration, then use human evaluation for final sign-off before release. Together, you get speed and safety.

Bottom line: automated evaluation is the AI's "machine examiner" — fast, cheap, and built for scale, but humans still make the final call.