Apps · PROJECT NOTES
Berkeley Function Calling Leaderboard (BFCL)
A function-calling evaluation suite combining static call-structure checks, executable simulated multi-turn tools, and agentic web-search and memory tasks.
What can you do with it?
Separate function selection and argument accuracy from multi-turn state handling, abstention, search and memory. Run the released generation and category-specific evaluation tools.
What should you know before using it?
Pin the v4 category set: single-turn/live uses AST checks, multi-turn checks simulated state and responses, and agentic tasks match answers. 'Live' means user-contributed examples, not live API execution. Memory/search variants reuse questions. Harbor's 3,641-task adapter excludes multi-turn and agentic tests; its 123-task parity subset adds no new tasks. Web search needs SerpAPI or a replacement. Older single-turn execution/REST/SQL categories are inactive; partial runs and weighted summary scores are not interchangeable.
What is verified here?
Runtime: Python >=3.10 package bfcl-eval, with bfcl generate/evaluate commands. Provider credentials or a local model backend are needed; vLLM/sglang are optional extras. Native device platforms are not asserted.
License note: The Gorilla repository LICENSE and bfcl_eval package declare Apache-2.0. External model/search services and dependencies retain their own terms. Upstream license ↗
This entry was reviewed against original project material. We have not independently run the software or reproduced its results. AI-development claims are author-reported; check upstream for current versions, setup requirements, and licenses.
- Resource type
- Benchmark suite
- Provenance
- Development provenance not established
- Reviewed
- Platforms
- Consult upstream documentation