benchmarks.wiki

benchmarks.wiki / Public workspace

GAIA

Agents / Multimodal

GAIA evaluates general AI assistants on tasks involving information gathering and tool use. Some tasks include attachments; this catalog provides an introduction in view of access and resharing restrictions.

gaia

Official source

Download answersVisit the official benchmarkWhere the answers areStart a benchmark discussion

History

The GAIA research group introduced the benchmark in 2023 to evaluate general AI assistants on tasks requiring information gathering and tool use. The dataset was published under the gaia-benchmark organization on Hugging Face in October 2023. Its tasks connect language understanding to retrieving information and handling supporting materials, including attachments.

At a glance

Released
Dataset published on Hugging Face 2023-10
Maintainer
gaia-benchmark (Hugging Face organisation)
Answer access
This catalog has not attempted authenticated acquisition; its acquisition record notes gated access and resharing restrictions.
Highest recorded
30.3 Score in % — GPT4 + plugins* (paper)

What a problem looks like

Information-gathering and tool-use tasks; some include attachments.

Reported scores

Evaluation conditions differ between rows, so this is not a ranking. Rows marked vendor are reports by the model’s own organization.

ModelScoreMetricSetupReported bySource
GPT4 + plugins*30.3Score in %Level 1 (146 questions). Plugins manually selected; source says results cannot be reproduced exactly.paperTable 4, Level 1 score
AutoGPT (GPT4 backend)14.4Score in %Level 1 (146 questions).paperTable 4, Level 1 score

Why it is hard

Requires gathering information and using tools, including tasks with attachments.

Limitations and criticism

GAIA scores reflect the complete assistant setup, including access to tools and the ability to process attachments, rather than the language model alone. The paper's recorded GPT-4 result used manually selected plugins, and the source explicitly states that those results cannot be reproduced exactly. A result on Level 1 also cannot be treated as an overall score across the benchmark's difficulty levels.

How much room is left

The recorded paper results on the 146 Level 1 questions report 30.3% for GPT-4 with manually selected plugins and 14.4% for AutoGPT with a GPT-4 backend. Those experiments left considerable headroom even at Level 1. They are historical results under specific tool configurations and do not establish how close later systems are to saturating GAIA.

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Public problem summary

Catalogue metadata only; public problem statistics are not available.

Public problems
—
Formats
—
Problems with images
—
Answer availability
—
Configs
—
Splits
—
Awaiting a first discussion
—
Problems with a discussion
—
Public contributions
—