Apps · PROJECT NOTES
ScienceAgentBench
An executable benchmark for generating standalone Python programs that perform data-driven scientific analyses, with task assets and direct or containerized evaluation.
What can you do with it?
Evaluate whether generated programs produce task-specific scientific outputs, using execution checks and visual judging where required.
What should you know before using it?
Use the verified split with matching benchmark_verified assets; it was announced after the frozen 102-task Harbor package. The public runner still defaults to validation, so select the release explicitly. Some outputs use model-based visual judging and results can depend on scientific-library versions or randomness. Full assets are separately hosted and were not downloaded; authors request no redistribution of unpacked files.
What is verified here?
Runtime: The documented setup uses Python 3.10 and Docker evaluation or per-task Conda environments. Provide the benchmark assets and configured judge/model access. Task-specific dependencies, compute and memory affect reproduction; no programs, images or scientific datasets were run.
License note: Repository code is MIT. Most task material is CC BY 4.0; rasterio- and matminer-derived tasks retain their original licenses. The README's unpacked-file redistribution request is recorded separately from those license declarations. Upstream license ↗
This entry was reviewed against original project material. We have not independently run the software or reproduced its results. AI-development claims are author-reported; check upstream for current versions, setup requirements, and licenses.
- Resource type
- Environment + benchmark
- Provenance
- Development provenance not established
- Reviewed
- Platforms
- Consult upstream documentation