Benchmarking

What is benchmarking?

Model A says “I'm great”, model B says “I'm great too” — how do you compare them? A benchmark is a public, standardized exam: fixed questions, fixed scoring, and anyone can run their model through it to get a number you can read at a glance.

What are the common benchmarks?

Knowledge Q&A
MMLU, for example, quizzes everything from physics to law, measuring how much a model “knows”.
Reasoning and math
Sets like GSM8K and MATH test whether a model can “think” with math and logic problems.
Coding
HumanEval, SWE-bench and friends check whether a model writes code that actually runs and fixes real bugs.
General-purpose suites
Many teams also use in-house eval sets covering long context, multi-turn chat, tool use and more.

How to read a leaderboard without getting burned

Watch for “studying the test”
Some models saw the exam during training, which inflates their scores — that's data contamination. Keep that in mind.
A high score isn't the same as useful
A benchmark measures one slice; real tasks vary wildly. In the end you have to test on your own workload.

Why it matters

Benchmarks give the whole industry a shared language: vendors use them to claim progress, researchers to locate gaps, and you to quickly filter out options that won't work.

Bottom line: benchmarking is the AI's standardized exam. Scores are a useful signal — but never the only one.