Apps · PROJECT NOTES
ToolHop
A multi-hop tool-use benchmark pairing questions with interdependent local Python tools, tool documentation and reference answers.
What can you do with it?
Compare direct answering, mandatory tool use and free tool use on the same query set, using released local-model or API-model evaluators and execution feedback.
What should you know before using it?
Authors report 995 queries and 3,912 associated tools; tools are not independent task counts or live commercial APIs. The scorer fixes its denominator at 995 and can credit an answer found in the last tool response, so partial-run output and strict final-answer accuracy differ. The supplied evaluator executes tool snippets and parses answer strings with eval in the host process; no isolation layer was established. Code is Apache-2.0 and data CC BY 4.0.
What is verified here?
Runtime: Public ungated Hub repository with data/ToolHop.json, evaluation_open.py/evaluation_closed.py and requirements.txt. Local inference uses Transformers; hosted inference requires model access. No large corpus file was downloaded or evaluator executed in this review.
License note: The official card separates Apache-2.0 code from CC BY 4.0 data. The root LICENSE is Apache-2.0; the dataset's CC BY declaration is in the card. ToolHop derives its questions from MorehopQA, whose provenance should remain attached. Upstream license ↗
This entry was reviewed against original project material. We have not independently run the software or reproduced its results. AI-development claims are author-reported; check upstream for current versions, setup requirements, and licenses.
- Resource type
- Environment + benchmark
- Provenance
- Development provenance not established
- Reviewed
- Platforms
- Consult upstream documentation