Apps · PROJECT NOTES
SWT-Bench
A software test-generation benchmark that asks agents to turn repository issues into tests that fail before a repair and pass afterward.
What can you do with it?
Evaluate issue-reproducing tests using Docker-backed before/after checks, applicability and coverage-change reports.
What should you know before using it?
The issue families derive from SWE-bench; Lite, Verified and Full are filtered variants, not independent task universes. Unit-test mode and standalone reproduction-script mode have different checks; script mode uses exit status and does not run the suite. Upstream lists 276 Lite, 433 Verified and 2,294 Full instances. Pin the dataset and filters. Harness MIT does not replace the licenses of evaluated repositories. No Docker build, generated test or coverage result was reproduced here.
What is verified here?
Runtime: Python 3.9+ and Docker. The published guidance recommends x86_64, at least 120 GB free storage, 16 GB RAM and 8 CPU cores for fast evaluation; arm64 support is experimental. Preserve the execution mode and available gold evaluations when reporting coverage deltas.
License note: Benchmark harness code is MIT. SWE-bench-derived task content, original repositories and dependencies retain their separate terms. Upstream license ↗
This entry was reviewed against original project material. We have not independently run the software or reproduced its results. AI-development claims are author-reported; check upstream for current versions, setup requirements, and licenses.
- Resource type
- Benchmark suite
- Provenance
- Development provenance not established
- Reviewed
- Platforms
- Consult upstream documentation