benchmarks.wiki

benchmarks.wiki / Public workspace

ARC-AGI-2

Reasoning

ARC-AGI-2 evaluates the ability to infer unfamiliar rules from a few examples and apply them to new grids. Training and evaluation are separate; this import covers the 120 public evaluation tasks.

arc-agi-2

Full text120 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

ARC Prize introduced ARC-AGI-2 in 2025 to test whether systems can infer unfamiliar rules from a small number of demonstrations. It continued the ARC approach of using abstract grid transformations to study generalization beyond familiar task templates; the source repository was created in March 2025.

How it was built

Tasks consist of demonstration input-output grid pairs followed by test inputs requiring the same underlying transformation. Grids use ten integer color values and range from one to thirty rows and columns. The catalogue selection contains 120 public evaluation tasks, with training tasks in a separate split and supplied test outputs stored separately from prompts.

At a glance

Size
120 public evaluation tasks in this catalog selection; training is a separate split.
License
Apache-2.0 (declared in the GitHub repository)
Released
Repository created on GitHub 2025-03
Maintainer
arcprize (GitHub organisation)
Answer access
Training outputs are examples in the prompt. Test output grids, when supplied, are separated into restricted answers; missing test outputs are allowed.

What a problem looks like

Each task contains train input/output grid pairs and test input grids. Grids are rectangular, 1–30 rows and columns, with integer colors 0–9.

Read an actual problem

How it is scored

For the 120 imported problems, the source ships an answer with every problem; this catalogue keeps it in restricted storage and never renders it.

Why it is hard

Requires inferring unfamiliar rules from a few examples and applying them to new grids.

Limitations and criticism

Success concerns a constrained world of colored grids and does not establish competence in language, physical interaction, or extended real-world tasks. The public evaluation tasks include accessible target outputs in the source, so restricting answers in this catalogue does not make the underlying benchmark private. Results on these public tasks should therefore be distinguished from results on held-out evaluations.

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
120
Formats
grid (120)
Problems with images
0
Answer availability
Answer published by the source (120)
Configs
Default (120)
Splits
evaluation (120)
Awaiting a first discussion
119
Problems with a discussion
1
Public contributions
1

Provenance and terms

Pinned source revisions, newest first:

  • f3283f727488ad98fe575ea6a5ac981e4a188e49 · acquired 2026-09-09 · release d292864f8da6
  • f3283f727488ad98fe575ea6a5ac981e4a188e49 · acquired 2026-09-09 · release e20b468c97b2
  • f3283f727488ad98fe575ea6a5ac981e4a188e49 · acquired 2026-09-09 · release 199f8507c15a
  • f3283f727488ad98fe575ea6a5ac981e4a188e49 · acquired 2026-09-09 · release 69480bfd3a14
  • f3283f727488ad98fe575ea6a5ac981e4a188e49 · acquired 2026-09-08 · release dce142b1444c
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never