Apps · PROJECT NOTES
BixBench
A bioinformatics benchmark with a released notebook execution environment for agents that inspect data, run Python/R/Bash analyses and interpret results.
What can you do with it?
Evaluate multi-step computational analyses with open answers or multiple-choice answers, separately from zero-shot question-answering baselines.
What should you know before using it?
v1.5 revises and flattens the benchmark to 205 questions; v1.0 has 296 questions and is not score-equivalent. The public split is named train but includes benchmark answers and a no-training-corpus notice. Open answers can require model judging. Capsule counts differ across paper, README and current metadata. Harbor notebook/CLI variants alter tools, prompts and grading details.
What is verified here?
Runtime: Python >=3.12 with Aviary/fhda and a Jupyter kernel; upstream recommends its Docker image. Data capsules and model/judge access are separate requirements. Hub metadata is public and ungated despite the README login instruction. Pin dataset revision, execution mode, grader and replica settings.
License note: Repository code and the public dataset card declare Apache-2.0. Third-party capsule files were not unpacked or individually licensed in this review. The dataset contamination notice is an evaluation-integrity instruction, separate from the license declaration. Upstream license ↗
This entry was reviewed against original project material. We have not independently run the software or reproduced its results. AI-development claims are author-reported; check upstream for current versions, setup requirements, and licenses.
- Resource type
- Environment + benchmark
- Provenance
- Development provenance not established
- Reviewed
- Platforms
- Consult upstream documentation