benchmark
SWE-bench
A benchmark for evaluating language models on real software engineering issues from open-source repositories.
What it is
SWE-bench pairs GitHub issues with repository snapshots and tests to measure whether generated patches resolve real tasks.
Benchmark evidence
SWE-bench maintainers
Evaluated target: Published leaderboard submissions
- Primary measure
- Resolved task percentage
Limitation: Results depend on the benchmark variant, execution environment, and submission policy.
Methodology Results