Discussion
0 posts
No posts in this thread yet.
benchmarks.wiki / Public workspace
.·:-=+**++-:·. ..··:-----------------========---:::·········..... ..·:-=*%@@@@@@@@@%*+=::·····::-=++***++=-:·.
.··:-----::··.........······..........···::---------::···.............····:-=+#%@@@@@@@@@@%*+-:······:--=+*#**+=-:·.
...·····............... ...··········.... .···::--=+*#@@@@@@@@@@@@#*=-:··..··::-=+**++=-:·.
...·:-===++*%@@@@@@@@@@@@%#+-:·......·::-====-:··.
..::=+++**#%@@@@@@@@@@@@@#+-:·. ..·:::::··..
..:-+****##%@@@@@@@@@@@@@#+-:·. ...···...
..:-**####%%%@@@@@@@@@@@#*=-·. ......
..:-**##%%%%%%%@@@@@@%%#*+-:·. .....
.::+*##%%%%#######***++=-:·.. ......
..···::::::::···.... ...·······... .·:+*##%%###**+++==---::··... ....···...
..·:-=++*****++=--::··.... ..··::-----::·.. .·-=+*#####**+==-::···..........···:::---::··..
.·:=+#%@@@@@@@@%#*=-::·.... ...·::-=++++=--:·. .·:-=+*###**+=--:·... ...··:-==++***+=-:·..
.··:::::··......··:--==++++==---:::···:::::::::::::::::::···.. ·:=*#%@@@%%#*+=-::·······:::---====--::·..
...···········:::------::···... ....·············......... .·-=*%@@@@@@%#*+=-:·······::--=+++*+==-:·.
........·········.... ............ .:-+#%@@@@@@@@%*+=-:······::--=++**+==-:·.
..·:=+#@@@@@@@@@@%*+=-:······::--==++==-:·..
..··:-=*#@@@@@@@@@@@%*+=-:··...····:::-::··...
..··:--=+*%@@@@@@@@@@@@#*=-:·... .............
..::===+*#%@@@@@@@@@@@@#*=-:·.. ...
··--++***#%%@@@@@@@@@@%#+=-:·. ...
···==########%%%@@@%%%#*+=-::·.. ....
...···::::::::····.... ...·········...........···===%####*******++++===--::······....····......
..·:--=++++++++====---:::··... ...·::-======--::······::--===%%%###*++==--::::·······::::---=====--::·...
.·-=*#%@@@@@@%##*+++===----::···.......·:-=+*###**+=--:::::--=+*###%%##*+=--:··.. ...··:-=++*######*+=-··.
.. ..·:-=+**######**++==--:::::··········:::-----::·. .·:-=+****++===---:::::::::::::::---::··...
..··:--==++++==--::···......................·····.. ·:-=+**#**++==--:::···::::--========-:··..
...···::::::···... ·:-+*######*+==-:::···:::--==+++++++=:··.
·:-+*#%%@%%#**+=--:::::::--==+++++==-:·..
.·:=+#%@@@@@@%#*+==--::::::--------::··...
·:-=+#%@@@@@@@@%#*+==--::::::·····.....
..::-+*#%@@@@@@@@@%#*+==-::···.... ....
..··-==+*#%%@@@@@@@%%#*+==-::··... ....
··.--++++**###%%%%###**++==--::···..... ......
....·············.... .....····.............········===***+++++++++=======------------::::···.....
..·:::------=========--::··......··::-------:::·····::-==+++***++%##*+==--::········::--==+********++=-:··..
.·:-=+*****+++++++****#***++=-::·····:--=+*****++==----=++*#%@@@@@@@@%#*+-:··. ..·:=+*#%@@@@@@@%#*+-:·.
.·:-=+*#%%%@@@%%%###**++=--::·····::-=+*****+=-:·. .··:-====+=================----------:::··..
..·:--==++++++===------::::···.........··::---::··. .·--==+++===----:::::::---===+++++++==-:·..
..···::::::·····.......·...... ....... ..·-==++++==--:::·····::--==+***##**++=-:·.
.·-==+++++==--::·····::--=++***#**++=-:·...
..:-=++**+++==--::::::--===+++++==--:··......
..:-=++*****+++===-----=======--::··...........
.·--=+***###***+++++======---::·... .......
··-==++***********+++++===--::··.. ....···..
...::===+++++++++++++++++++===---::···········....
........·....... ..... ...··········---+====-----:::----===++++++++++===---::···..
.....······::::-----:::··.......········... ..·::-=++******++=**+=-:··........··:-=+**#%%%@%%%##*++=-:·..
..··:::--::::::--=+++***++=--::···:::------::··.....·:-=+#%@@@@@@@@@@@#*=-:.. .·:=+#%@@@@@@@@@@%#*+-:·.
.·:-==+*##%%@@@@@@@@@%#*+==-----=++*#%%%%#*+=-::······:--===++++==================++++++++++==--::··..
..·::---======++++++++++==--::·····::--=====--::·........··::----+++===--:::·····:::--=++**####**++=-::·..
..······················=··..... .....····.... .....---*++=--:··.......··:-=+**#%%%%%##*+=-:··..
...--+++=--:·... ..·:-=+*##%%%%##*+=--:··.....
..--+++=--:··.......·::-=++******+==-::::·····...
..::=====--::······::--==+++++==-::········::···.
..::=====-----::----===+++===-::··.....··::::::·.
..::=====-------====+++++===--:········:::::::··.
...::=-------------===+++++===----::::------::··..
........ ...... ....··········-::=--::········::--==+++*****++++++++++=-:··.
..........······..... ....·······.. .·::-=++*****++==-=-:··. .··:-=+**#%%%%@@%%%###*+=-:·.
...·........···::::::····..···:::---::·. .·-=*%@@@@@@@@@@@%*+-·. .:-=+*%%@@@@@@@@@@%#*+=:·..
.··::-=++*#%@@@@@@%#*++======++**##**+=-::··..··::-==+++***+++====-------------==+++*******++==--::::··....
...·····:::--=+++*****++=--::::::---===--::··.......··::-=========*+=--:···......··:-==+*##%%%%###*++==-::·...
............····::::::::·=··.......······.... ....····::::+==+=-:·.. .·:-=+*#%%@@@%%##**+==-::·..
::·==*=-:·. ..:-=+*##%%%%##**+++===-::·..
··==+=-:·. ..·:-=++****+++==========--:·..
··--+=-:·... ..··::-======---::::::--=++==--:·.
··-:=--::······:::--=====--::·······:--=+++==-:·.
..::--::::::::---=======--::······::--=+*++==-:·.
...::-:::::::::---=======---:::::---==++**++=-:·..
........ ....·····... .....·········.::·::·········:::---========++++********+=-:·..
.....····..... ..··::----::·. .·::-==++**+++==-:::··.. ...·::--==++**###%%%%%%%##*=-:·.
.......······.... ..·:-==+++=-:·. .·-=*%@@@@@@@@@@%*+-:·. ..·::-=++*##%%@@@@@@@%#**=:·.
..··:--=++++++===-----==++**++=-:·. .··:-==++*****+++==---:::·····:::---==++++****++++++++++=-::·..
....··::--====---:::::::------::··. .··:-==+++++++==-+=-:·.. ..·::-=+**###%%%%%%####*+=-:·.
...···········..-...·····.... ...··:::::::::+=-=-·. .·:-=+**##%%%%%%%%%%##*=-:·.
:::=-=:·. .·:-==++****######%%##*+=:·.
:·=--:·. ..·::-----======++**#####+=-:·.
··-:-:·... ...···:::::::::····::--=+*##%#*+=-·.
·.::-::····:::-------::···.....··:--=+*#%#*+=-:·.
..:·:::::---=======--::··......··:-=+**#%#*+=-:·.
...··::::---========--::·······::--=++*###**+=:··.
....··... ...··:::::::··... ......······.....···:::::::---------:::::::---==++***###*+=--:·.
..··::::::·... ..·:-==+***++=-:·.. .··::-========--::······...······:::::::---==++**########**=-:··..
.··:---=--:·.. ..·:-=*#%%@%%#+=:·. .·:=+*%@@@@@@@%#*+=-:··... ......···:::---==+**##%%%%%##*++-:·..
..···::::······::-=+*##%%#*+=-·. ..·:--=++*******++==--::···········::::---==++**##%%@@@%%#+=:·..
....············::--=====-:·.. .··:-=++*****++==-=-:·.. ..··:--=+**#%%@@@@@@@@%*+-:..
..............·-·····... ..··:::::::::::=-::·. .··:--=+**#%@@@@@@@@@%*=-·.
::·-::·. ..··::--==+**#%@@@@@@@%*=-·.
··:·:·. ....········:::-=++*#%%@@@%*+-:.
·.:·::·····::::::::···.......·::-=+*#%%%%#+=:·.
..··----==+++++==--:·.. ..·:-=+*#%%%#*=-:..
..··-==+**####*++=-:·.. ..·:-=++*#%%#*+-:·.
.··==+**####**+=-::·.........·::-==+**###*+-:·.
....... ...···:::::::····..... ............... ···-==+++++++==--::········::---==++*****+=-:·..
.··:::::·......·:--=+******+==-::·... ...··:::------:::·········::----------::::::::::---===+++++++++++=--:·..
.·:-=++==-:·...··:=+*%@@@@@@@#*+=:·.. ..·:-=+**####**++==-:::::::::::::::::::::::::--==+++******+++===--::··..
..···::::···....··:-+*%@@@@@@@@#*=:·. .··:-=++**####**++==--:::····.........··::-=+*#%@@@@@@@@@#*=-··.
....·...... ...·:-=+******+=-:·. .··:-=++*****++==-:-:··.. ..··:-=+*#%@@@@@@@@@@%*=-·..
..···:::*:··.. ..··::::::::::··-:··.. ..··:-=+*#%@@@@@@@@@%#+-·.
··.···... .....···:-=+*#%@@@@@@@@%*=:·.
..··::············......··:-=+*##%@@@@%#*=:·.
..··-===+++++==-::·... ..··:-=+*##%%##*=:·.
..::+*##%%%%##*+=:··. .··:-=+**##**=-·.
..:-#%@@@@@@@%*+=-:·. ..·:-=++****=-:·.
..:-#%@@@@@@%#*+=-:·.. ..··:-==++**+=:·..
.....·········............................ .::-*###%###*++=-::···........···::-==++*+=-:·.
..·.........··::-========---:::··········::::------:::··.... ...···:=+++++++===---::::::::::::::::::--==+++=-::·.
.·:::::······:-=+**#%%%%%##*+=--:········::--=+++***++==--::······:::---=====-----::::-------=------::::--==+==--:·.
.·:----::·....·:-+#%@@@@@@@@@@@%*+=:··. ..·::-==+++*****++++====--::···... ...·::-=+*#%@@@@@@@@@%*+=-··.
..·······.......·:-=+*********+=-:·. ..·::-==++++++===--::-::··.... ..·::-=*#%@@@@@@@@@@%#*=-:··.
...... ..··::::::::*··.. ..···::::::::···..:·····.... ..··:-=+#%@@@@@@@@@%#*+=-:·..
...··::······.......··::-=+*#%@@@@@@@%#*+=:·..
..::-======--:::·······:-==+*#%@@@@%#*+=-·..
..:-*######**+=-::··..···::-=+**####*+=-:·.
..-=%@@@@@@@%#+=-:··......··:-==+++++=--·.
.·=+@@@@@@@@%#*+=:··.. ...·::-=====-:·.
..·=+@@@@@@@@%#*+=-:·.. ...·::-----::·.
......... .........······::····... ..--=%%%%%%##**+==-::··.. ..··::---::·.
...····:::::::··············::--==++++++==-:··.. ..··:::+++**+++++====---::··... ...·:::--::·.
.......··::-===+++++==--:::·····:::-==+*###%%##*++=-:··......···::----==---------=======--::··.......··::---::·.
..····.......·-=+#%@@@@@@@@@@@%#*+==-:··........···:::-----========++++====--:···........··::-==++**###%%@@@@%#*=-:..
..··.........·:-=+**********+=--:·.. ..··:::-----------:::::·-:::··.........··:--=+**##%%%%%%%%%%%%#*+=-:·.
.... ...··::::::::::·=.. ....···········......···:···.........··:-=+*#%%@@@@@%%###****++=-:·..
...··:::::········::-=+**#%@@@@@@%%#**++===--::·..
..::=++====---::::--==+*#%%@@@@@%#*+==---:::···.
..-=##%%##**+==------==+**##%%%##*+=--:····.....
.·=+@@@@@@%%#*+==-------==++*****+=-::·... ...
··+*@@@@@@@@%#*+=--::::::---======-::·..
.··++@@@@@@@@%#*+=--:·······:::----::·..
........ ...··::::::::::··.. ....-==%%%%%%%%##*+==-:··......····::··..
..···::::···.... ..··:-=+********+=-:··. ...···::::++++********++=--:·... ....··...
..··:-======--::··... ..··:-=+*#%@@@@@%#*+=--:········::::::--:::::----==+++++++=-::·.. ......
..·:-=+**##%%%%%%###**++==----::::---------::::::::::----======---:::··········::::::::---==+**#%%%%*+-:·.
..··:-==++*****++==--::··.........···:::::--:::::::::::·······:····.....···::--=============++**##%%*+=:·.
...···::::::::···..· ....·········........... ...·.......··::-==+**####**+++===+++**##*+=-:..
..·········::-=+*##%%@@@%%#*++=====++***+=-:·.
..··----::::---=+**#%@@@@@@@%#*+==---==++++=-:·.
..--+++++======++*##%@@@@@@@%#*+=--:::--===--:·..
··==#%####***+++***#%%@@@@@@%#*+=-:···::-----:··.
··++@@@@@%%##*******###%%%%%#*+=-:··...··:::::··.
.··=+@@@@@@%%##***++++****##**+=-:··.. ...·····..
.......... ..··::::::::::··.. .....-==#%%%@@%%%##*+++=====++++==--:·. ..···..
..·::----:··.. .·:-=+*******++=-:··...........·······::::++**##%%%###*++==----------::·. .....
.·:-==+*++=--:·. .·:-=+*#%%%%%#**+==----------===----::::::::--=+**#####*++=--::::::::···.. ....
..·::---===========----------==+++******++==--:::·····:::::::::::::::::::-------::::······::--==+++==-:·.
..·::---=====---::···........···::----==-----:::::········...........···::--======--:::···::--======-:·..
...·····:::····... ...············......... ..·::-=++****++=--:::::::--====--:·.
..·:-=+*##%%%##*+=-:::::::--====-:··.
. .....·::-+*#%@@@@@@%#*=-:::··::--====-:··.
·.····::--=+*%@@@@@@@@%#*=-::···::--===--:··.
..::-----===+*#%@@@@@@@@@%#+=-:·····::-===--:·..
..--+++++++**##%@@@@@@@@@@#*=-:·...··::-===--:·..
.··-=#####**###%%@@@@@@@@@@%*=-:·.....·::-==--::·.
...····... ..··:::::::::··... ......·-==###%######%%%@@@@@@@@@#*=-:·. .··:-----:·.
.··:-----::·. .·:-==+++++==--::······················:::--*##%%%%%%%##%%%@@@@@@@%#*=-·. ..·:----:·.
.·:-=+***+=-:·. .··:-==++++===--:::::--==+++****+++=---:::---=+**##%%%%%######%%%%%%#*+=:·. .·:::::·.
..·:--======--::···.......··:-==+*#%%%%%%##*++==--:::::::::·········:::--===++++==---:::---==++++==-::·..........
..·::--===---::··.. ...·::--==========-----:::::···.... ...·::-==+++++==--:::::---===--::·..
..····::····... ....················...... ..·:-=+*****+==-::::::::---::··..
.·:-=+*####**+=-::·····::::···..
.·:-=*#%%@%%#*+=-:············..
.·:-+*#@@@@@@%*+=::·......····....
..·:=+*%@@@@@@@%*+-:·.........···......
...···:-=+#%@@@@@@@@%*=-:·.. ...········....
...··::::-=+*#@@@@@@@@@@#+=:·. ..··:::::::··..
...··... ..··:::::····.... ...........:::=-===++*%@@@@@@@@@@%*=-·.. ..·:------::·.
.··::--::·. .··:---=----::··........····:::::::····::::::---+++++++*#%@@@@@@@@@@@#+-:·. .·:-=====-:·.
.·:-=====-:··......··:--=====--:··... ...·::-=+***##**++===----===++*********#%@@@@@@@@@@@%*=:·. .·:-=====-:·.
.·:-=++*#**+=-:·. .·:-=+*##%%%%%##***+++====----::·········::--==++*********##%%@@@@@%#*+=-:·····:--==--:·.
..·:-==++==--:·.. ..··:::-----============---::··.. ..·:-=++***********###%%%%##*=-:··.....·::::::·.
..···::::···.. .......···············.... ·:-=+**####*********##**++=-:·. ..·····..
.·:-=+*#####***+++++++++==-:·. ......
.·:=+*#%%%##**++=====---::·.. ....
.:-+*#%%%%##*+==--:::::··.. ..
.·-=*#%@@@%#*+=--::····... ..
.·:=*#%@@@@%#*+=-:·..... ....
.·:=*#@@@@@@%#+=-:·.. .........
..... ...·········.... .......... ..·-=*#@@@@@@@%*+-:·. ...··::::···.
..·····.. ..·::--====--:··... ....···········:::::····.......·:-=*#@@@@@@@@#*=-·. ..·:---=---:·.
.···:::::··:::-==+**###**+=-:·. .·::-===================--:::···::-+*#@@@@@@@@@#+-:·. .·:-=++++=-:·.
.:-=+*###*+-:·. .··:-==+++++++++++++++++++===--::········:::----====++*#%@@@@@@@@@@@%*=-::··::-=+*##**+=-·.
..·:-==++==-:·.. ...········::::---========--::··.........··-==++++**#%%@@@@@@@@@@@%*+-:·····::-=++++==:·.
...···::···... . .....····:::::····... .··=++***###%%@@@@@@@@@@@%*+-:·.....··:-====-:·.
··==+**###%%%@@@@@@@@@@@%*=-:·. ..·::::::·.
.·-=+*##%%%%%%@@@@@@@@@%#+=:·. .·····..
.:-+*##%%%%%%%%%%%%%%#*+=-:·. ......
.·-=+*#%%%%%#####****++=-:·. .....
·:-+*#%%%%%##**++===--::·. .....
.·-=*#%%%%%#**+==-:::··.. ......
..···:::::···.... .......... .·:=+#%%%%%#*+==-:··... ..······..
..·:-==++++++=--:··.. .....···::::::··.. .·:-+*#%@@%%#*=-:··. ..··::----::··.
...··:-=*#%@@@@@@@#*+-:·. ...······:::---====--:·.. ..:-=*#%@@@%#*+=:·.. ..·::-=+*++=--:·.
█████╗ ██████╗ ██████╗ █████╗ ██████╗ ██╗ ██████╗
██╔══██╗██╔══██╗██╔════╝ ██╔══██╗██╔════╝ ██║ ╚════██╗
███████║██████╔╝██║ █████╗███████║██║ ███╗██║█████╗ █████╔╝
██╔══██║██╔══██╗██║ ╚════╝██╔══██║██║ ██║██║╚════╝ ╚═══██╗
██║ ██║██║ ██║╚██████╗ ██║ ██║╚██████╔╝██║ ██████╔╝
╚═╝ ╚═╝╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═════╝ $ open /benchmarks/arc-agi-3▌highest recorded ░░░░░░░░░░░░░░░░░░░░ 0.37 Score (%) · Gemini 3.1 Pro Previewready
ARC-AGI-3
Reasoning / Agents
ARC-AGI-3 is an interactive benchmark for exploring unfamiliar game environments, learning their rules, and planning actions. The official toolkit provides an environment interface, with game versions and trial records tracked separately.
arc-agi-3
At a glance
- Highest recorded
- 0.37 Score (%) — Gemini 3.1 Pro Preview (paper)
What a problem looks like
Interactive game environments accessed through an official toolkit, with game versions and trial records tracked separately.
Reported scores
Evaluation conditions differ between rows, so this is not a ranking. Rows marked vendor are reports by the model’s own organization.
| Model | Score | Metric | Setup | Reported by | Source |
|---|---|---|---|---|---|
| Gemini 3.1 Pro PreviewGoogle | 0.37 | Score (%) | Semi-private set; official leaderboard at release; same system prompt, no tools supplied by the evaluator. | paper | Table 2; Section 4.3.1 |
| GPT 5.4 (High)OpenAI | 0.26 | Score (%) | Semi-private set; official leaderboard at release; same system prompt, no tools supplied by the evaluator. | paper | Table 2; Section 4.3.1 |
| Opus 4.6 (Max)Anthropic | 0.25 | Score (%) | Semi-private set; official leaderboard at release; same system prompt, no tools supplied by the evaluator. | paper | Table 2; Section 4.3.1 |
Why it is hard
Requires exploring unfamiliar game environments, learning rules, and planning actions.
Limitations and criticism
Performance depends on interaction with a game environment, so the score combines rule discovery, exploration and action planning rather than isolating any one of them. Comparisons need to account for game versions and evaluation conditions, including which tools are supplied. The recorded release results use a semi-private set and a shared system prompt with no tools supplied by the evaluator; they do not establish performance under other agent configurations.
How much room is left
The recorded official release leaderboard reports scores of 0.37% for Gemini 3.1 Pro Preview, 0.26% for GPT 5.4 (High) and 0.25% for Opus 4.6 (Max) on the semi-private set. These reported scores are far from the top of the percentage scale and provide no indication of saturation in that setup. They are benchmark scores, not evidence that the same percentages of complete games were solved.
Reference links
- Site
Related benchmarks
Discussion
1 thread and 0 posts on this page. Anyone may reply; no account is required.
Contribute
Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.
Public problem summary
Catalogue metadata only; public problem statistics are not available.
- Public problems
- —
- Formats
- —
- Problems with images
- —
- Answer availability
- —
- Configs
- —
- Splits
- —
- Awaiting a first discussion
- —
- Problems with a discussion
- —
- Public contributions
- —