Apps · PROJECT NOTES
ToolTalk
A conversational tool-use benchmark with executable simulated APIs, local database state and annotated reference calls across everyday services.
What can you do with it?
Evaluate tool sequences after each user turn, including calls that change simulated calendars, messages or accounts. Inspect conversation success, call precision/recall and incorrect-action rates.
What should you know before using it?
The release has 28 easy and 50 hard conversations, not 78 tools; the tool inventory has 28 tools in seven plugins. Before each assistant turn the evaluator resets state and replays reference history, so model errors do not freely accumulate across the whole conversation. Success requires all reference calls and no unmatched successful state-changing calls. The predictor uses legacy OpenAI SDK interfaces with unpinned openai; compatibility and historical model availability were not tested.
What is verified here?
Runtime: Python source package 1.0.0 with local JSON databases, evaluator scripts and an extensible predictor interface. API calls execute simulated implementations; there is no native mobile or desktop target. Example shell scripts are under scripts/, despite shorter paths in the README.
License note: The repository publishes an MIT license covering its released code and associated materials; model API access has separate terms. Upstream license ↗
This entry was reviewed against original project material. We have not independently run the software or reproduced its results. AI-development claims are author-reported; check upstream for current versions, setup requirements, and licenses.
- Resource type
- Environment + benchmark
- Provenance
- Development provenance not established
- Reviewed
- Platforms
- Consult upstream documentation