LLM-as-a-Judge

What is LLM-as-a-Judge?

After you upgrade a model, how do you know it actually got better? The most trustworthy way is to have humans score answers one by one — expensive and slow. LLM-as-a-Judge flips that: a “judge model” evaluates the candidate model's output, grading it against criteria you set in advance, writing feedback and deciding what's good or bad.

How does it judge?

Pairwise comparison
Put the answers from model A and model B side by side and ask the judge which is better — ideal for “did the new version regress?” checks.
Single-answer scoring
Assign a score or rating to one answer: relevance, accuracy, helpfulness, and so on.
Rubric-based evaluation
Hand the judge a detailed scoring rubric and have it check each criterion, cutting down on arbitrary calls.

What makes it attractive?

Fast and cheap
A human takes minutes per answer; a judge model takes seconds and costs orders of magnitude less.
Scalable
Running thousands or tens of thousands of evaluations a day stops being a problem.
Reproducible
Fix the prompt and parameters and results stay basically repeatable — great for regression testing.

The catch

Judges have biases too — they favor longer answers, prefer their own style, and miss factual errors. So it can't fully replace humans. The usual move: the judge does a first pass, humans spot-check, or several judges cross-score the same answers.

Bottom line: LLM-as-a-Judge makes an AI the examiner — fast, but don't treat its score as gospel.