Apps · PROJECT NOTES
SWE-bench Pro
Scale AI's long-horizon repository-repair benchmark, with released public task containers, issue specifications and patch-verification tooling.
What can you do with it?
Evaluate patches against task-specific failing and regression tests across Python, JavaScript/TypeScript and Go repositories.
What should you know before using it?
Current public V2 has 642 tasks; HARD-51 is a subset, and legacy v1 has 731. The paper's broader collection includes nonpublic tracks. Frozen Harbor packages still advertise 731 and are not proven equivalent to V2. V2's reported protocol keeps the agent phase offline and re-grades captured patches in fresh containers. Do not combine versions, subsets or namespaces into an additive task count.
What is verified here?
Runtime: V2 ships Harbor task directories and public linux/amd64 image references. The documented locked protocol uses Harbor with Modal's phase-specific network policy and a separate re-grader. Legacy local-Docker examples belong to v1. Container pulls, task execution and reference-patch claims were not independently tested.
License note: Current harness/tooling is MIT. Task instructions, patches and tests retain the upstream repositories' licenses, including copyleft where applicable. The older Harbor adapter's Apache-2.0 statement does not establish uniform rights for the current release. Upstream license ↗
This entry was reviewed against original project material. We have not independently run the software or reproduced its results. AI-development claims are author-reported; check upstream for current versions, setup requirements, and licenses.
- Resource type
- Benchmark suite
- Provenance
- Development provenance not established
- Reviewed
- Platforms
- Consult upstream documentation