SWE-bench

What is SWE-bench?

Open GitHub and you'll see issue after issue — someone reports a bug, someone asks for a feature. SWE-bench collects those real issues, pairs them with the code repo and the eventual fix, and uses them to test AI: give a model the problem description, and see if it can dig through the code, pinpoint the problem, and produce a patch that passes the tests.

How is it different from coding puzzles?

The problems come from the real world
These aren't made-up algorithm questions — they're problems people actually reported in real open-source projects like PyTorch and Django. The AI faces an entire codebase, not a forty-line toy function.
Scoring means passing tests
A patch doesn't just need to look right — it has to pass the project's own test suite. Only then is the bug truly fixed.

How does it test the AI?

Input
An issue description, plus the whole repository.
Output
A code change — a patch.
Judging
Apply the patch, run the relevant tests; if they pass, the problem counts as solved.

Why it matters

Before SWE-bench, "AI can write code" was hard to measure. SWE-bench gave us a yardstick that matches real work — going from "solves algorithm puzzles" to "fixes real projects". It's a leap in AI coding ability, and the leaderboard where every big model now fights for the top spot.

Bottom line: SWE-bench tests AI with real GitHub issues and only counts a win when the bug is actually fixed and the tests pass.