benchmarks.wiki

benchmarks.wiki / Public workspace

Berkeley Function Calling Leaderboard

Agents / Tools

The Berkeley Function Calling Leaderboard evaluates tool use. It works up from a single call to parallel calls, multi-turn dialogue, and recognising when no function should be called at all.

bfcl

Official source

Download answersVisit the official benchmarkWhere the answers areStart a benchmark discussion

History

The Gorilla team at UC Berkeley developed the Berkeley Function Calling Leaderboard to evaluate whether language models could select and invoke tools correctly. Its Hugging Face dataset was published in February 2024. The benchmark extends the evaluation of generated text to structured calls, including argument selection and cases in which no function should be called.

How it was built

The source distribution contains several thousand evaluation cases organized into tool-use categories. Cases pair user requests with callable function signatures and expected calls, with settings covering single calls, parallel calls, multi-turn interaction and abstention from calling a function. Evaluation compares abstract syntax trees against expected calls and executes calls where the category supports execution.

At a glance

Size
Several thousand evaluation cases across the categories; none imported.
License
apache-2.0 (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2024-02
Maintainer
Gorilla LLM, UC Berkeley (source organisation)
Answer access
No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.
Highest recorded
77.47 Overall Acc (%) — Claude-Opus-4-5-20251101 (FC) (leaderboard)

What a problem looks like

A user request with a set of callable function signatures; the expected output is the call, not prose.

How it is scored

Scored by abstract-syntax-tree comparison against the expected call, and by executing the call where the category allows it.

Reported scores

Evaluation conditions differ between rows, so this is not a ranking. Rows marked vendor are reports by the model’s own organization.

ModelScoreMetricSetupReported bySource
Claude-Opus-4-5-20251101 (FC)Anthropic77.47Overall Acc (%)BFCL V4; overall; FC.leaderboardOfficial BFCL V4 data, Overall Acc column
Claude-Sonnet-4-5-20250929 (FC)Anthropic73.24Overall Acc (%)BFCL V4; overall; FC.leaderboardOfficial BFCL V4 data, Overall Acc column
Gemini-3-Pro-Preview (Prompt)Google72.51Overall Acc (%)BFCL V4; overall; Prompt.leaderboardOfficial BFCL V4 data, Overall Acc column

Why it is hard

Getting the function right is not enough — the arguments, their types and the decision not to call anything all count, and the last of those is where most models lose points.

Limitations and criticism

A correct function call demonstrates success under the supplied tool schema, rather than reliable operation in every external service that might expose a similar tool. Abstract-syntax-tree matching and execution-based checks test different aspects of correctness, so an overall score can conceal category-specific weaknesses. Comparisons also depend on the benchmark version and whether a model uses native function calling or a prompted representation of calls.

How much room is left

The recorded BFCL V4 leaderboard reports overall accuracy of 77.47% for Claude Opus 4.5 with native function calling, followed by 73.24% for Claude Sonnet 4.5 in that mode. Gemini 3 Pro Preview is recorded at 72.51% using prompting. These results leave substantial overall headroom, but the different calling modes and the mixture of categories limit what a single ranking says about a particular tool-use workflow.

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Public problem summary

Catalogue metadata only; public problem statistics are not available.

Public problems
—
Formats
—
Problems with images
—
Answer availability
—
Configs
—
Splits
—
Awaiting a first discussion
—
Problems with a discussion
—
Public contributions
—