Apps · PROJECT NOTES
DS-1000
An execution-tested data-science code-generation benchmark covering 1,000 Python problems across seven widely used libraries.
What can you do with it?
Check generated code using functional test cases and task-specific code-form constraints, and report accuracy by library or problem type.
What should you know before using it?
This evaluates scoped code solutions, not complete application development or GUI navigation. The simplified release preserves problem content but changes packaging and removes the original infilling mode. It executes generated code and needs appropriate isolation; separate processes also matter for stateful libraries. Pin dataset and dependency versions when comparing original and Harbor runs.
What is verified here?
Runtime: The current environment file pins Python 3.10 and scientific-library versions. The released runner launches per-problem execution checks, plus test_string where supplied. The original-format branch and simplified test split are two protocols for the same benchmark family, not extra 1,000-task sets. No generated code was executed.
License note: Repository LICENSE and official dataset card declare CC BY-SA 4.0. Scientific-library dependencies retain their own licenses. Upstream license ↗
This entry was reviewed against original project material. We have not independently run the software or reproduced its results. AI-development claims are author-reported; check upstream for current versions, setup requirements, and licenses.
- Resource type
- Benchmark suite
- Provenance
- Development provenance not established
- Reviewed
- Platforms
- Consult upstream documentation