Discussion
0 posts
No posts in this thread yet.
benchmarks.wiki / Public workspace
..··::::::::···::::---===--::·. ·:=+*%@@@@%*+=:·. .··::--==++*****+=-:·. .··::::·.
...·····....····:::-----::··. .·:-=+++++==-:·.. ...···::--=====--::··. ..::---::·.
......·······... ..····::····... ....·········..··.. .·:-====--:·.
...... ..·:-=++**+=-:·..
. .....··:-=+*##%%#*+-:·.
..··:-=+*#%@@@@@%*=-:·.
.·:-=*#%@@@@@@@@#+=:·.
.··=+#%@@@@@@@@@@#=-·..
..·=*#@@@@@@@@@@@%+-:·.
..............·············..... ......·····........··········... .·:=+#@@@@@@@@@@@%*=:·.
.....·······:::::--========--::··.........···:::::::::::::--===+++==-:·.. .·-+*%@@@@@@@@@@#*=:·.
........····:::-==+*######*+=-::·... ....··:::::::::::::-=++*#%%%%#*+-:. .:-+#@@@@@@@@@%#+=-·.
..·:-=+**###*+=-:. .·:=+*####*++=-::·... .·:-=++*###*+=-:·. ..··::···..
.··:--=====--:·. .·::-------::····..... ..··:---==---:··. ..·:::::··..
..·········.. ................... ...······.... ..·::----::·..
..··:--=====-:··.
.·::-=+******+=:·..
..:-=+*#%%@%%#*+=:·..
··=+#%@@@@@@@%#*=:·.
·:-*%@@@@@@@@@@#*=:·.
.··-#%@@@@@@@@@@%#*=:·..
...................····::::::····................ ....··::::::::::··.. ·-=*%@@@@@@@@@@@%#+=:·..
...·······........··::-==++++++==---:::::::::····... ..··:-=+********+=-·. .:-+#@@@@@@@@@@@@%*+=:·...
..··::::··........·:-=++*#####**++=======-----:··. .:-+*%@@@@@@@@%*=:· .·:=*#@@@@@@@@@@@%#*+-:··..
.·-=*#%@@@%#+=:·. ..··::---------=----::··. .·:-==++++==-::·... ....··::::::··...
.·:--=====-::·. .....·········:::····... ..··::::::::··.. ..···:::::::··..
...······.... ............ ........... ..··:::----::··..
..·::--===--::·..
.·:--=++++==-::·.
..:-=+*****+=-:·..
··=+*##%%#**+=-·..
·::*#%%@@%%#*+=-:·.
..·:-*#%@@@@@%%#+=-:·.
.......... .....··············:::::····.. ..·::::::::::::··.. ..:-+*#%@@@@@@@%#*+=-:·..
.··::::::··......···::::-------====+++***++=-:·. .·:-+***********+-:·.. ..·-=+#%%@@@@@@@%#**+=-:·..
..·::--:::········:::-------===++*#%%@@@@@%#+=:· .·-+#@@@@@@@@@@@#*=-:·.. ..·:-=+*#%%@@@@@@@%%#**+=::·..
.·-=+*####*+=-:··... .............··::--=====--:··.. ..·::--====---:::······::--==++++*+++=-::·..
..·::----:::··... ................·········... ..···::::::··.... ..·:--==++**+++==::·..
.......... ........ ........... ..:--=+++*+++=--··.
..:-==++++++=-::·.
..:-=+++++==-:··.
.·-==+++++=-::·.
.·-==++++=--:·.
.··==++++==-:··.
...··-==+++==--:··.
............................··:::::::::::··.. ..·::::::::::::··... ..::--==++++==--::·..
.··:::::-:::::::····......··:-=+**********+-:·. ..·:=+***********+=--:··.........·:--===++++===--:··...
.·::--==+++===--::··.....··:=+#@@@@@@@@@@#+-:. .·-=*#@@@@@@@@@@@%#*++=--::::·:::::---====+++====--:::·...
.·:--====---:::······::······......··::---===---:··.... ....·::-===++++==--:::···:::-=+*#%@@@@@@@@@%#*=-·..
..········..........................···:::::···.. ..·:::----:::··.........··::*#%@@@@@@@@@%*+-:·.
...... ............ ....····.... .·:-#%@@@@@@@@%#*=-·..
.:-#%@@@@@@@%#*=-:·.
.:-*#%@@@@%%#*=-:·.
.::**#%%%##*+=-:·.
.·:++*****++=-:·..
··:==++++==-::·..
...··---===--::··..
...···::····... ..··:::::::::::··.. ...····:::::····.... ..···:::::::::···..
..·:--=+*++==-:·.. ..·:-+**********+=-:·.. ..··:--==++++++++==--:::···.........······::·····....
.·:-+*#%@@%#*=-:.. .·:=*#@@@@@@@@@@%*+=-:·.....··:-==+*##%%%@@%%%%##**++==---::::···············.........
.··:------::::::::::::::····....···::-==++**++==-::··.....··:-=++*##%%%##*+=--:·······:-=+*#%@@@@@@@@@@%*+-·.
...········........... ..··::-------:··.. ..·::-===++==--::··........·::-=%@@@@@@@@@@@%*+-·.
....... ...·······... ...·········... ...=++@@@@@@@@@@@%*=:·
.··+*@@@@@@@@@@@#+=:.
··+*@@@@@@@@@@%*+-·.
··+*@@@@@@@@@%#+=:·.
.··++@@@@@@@@%#*=-:.
....-=+@@@@@@%##+=-:·.
··::--*######**+=-:··.
...···:::··.. ..··::::::::::··... .............. ··--==+++++++==--::··..
.··:-==+**+=-··. ..·:-=+*********+=--::··........········::::···········.........···:-----------:::····......
·:-=+*#%@#*=-:·.....··:=+*%@@@@@@@@@%#**+==---::::----------------------------::::::::::::::::::············.......
.··::--=========---::··.. ..·:-=+*##%%%%%#*+=--:······:-=+*#%@@@@@@@@%#*=-:·. ..·-=+*#%@@@@@@%#*=-·.
...··::::::::::··.. .··:-==++++==--:··... ...·::-=++******++=-:·.. .··:+#%@@@@@@@@%#+=:.
............ ...····:····... ...···:::::::···... ..:-=@@@@@@@@@@%*+-·.
..·=+@@@@@@@@@@%*=:·
··+*@@@@@@@@@@#+-:.
·:**@@@@@@@@@%*+-·.
··:**@@@@@@@@@#*=:·.
..···:***@@@@@@@@%*+-:·.
..-=++**@@@@@@@@%#*=-:·.
...···:··.. ...···········...... .......·==%%@@@@@@@@%#**+=-:·...
.··:-====-:··.......··::--====+====---:::·········........... .......····:::----=###%%%%###**+==--::··....
.·:-=+++++-::·····::-=+*##%%%%%%##**++===------------:::···.... ...···:::--===++++********++++===----:::···...
..·:-=+##%%%%##*+=-:·.. .··:-+*#%@@@@@@%#*+=--:·····::-=+*%@@@@@@@@@%#+-:· .·:=+*%@@@@@@@%#+-:·.
..·::=+++++==-:··. .·:--=++****++=--:··......··:--=+*********+=-:·. .:-+*%@@@@@@@%*+-:..
..··:::····.. ...···::::····.... ....···::::::::::··.. .·:+*#@@@@@@@%#+=:.
:-*#@@@@@@@%#+=:·.
..-=%@@@@@@@%#*=-·.
..==@@@@@@@@%*+-:.
.··++@@@@@@@%#+=-·.
...···++*@@@@@@%#*+-:·.
.·-==++*@@@@@@%#*+=-:·..
......... ........... ................... .........···=+%@@@@@@@%%#*++=-:··..
.···:::···.......····:::::::::····.........···············................···:::---===+++@@@@@@%%##*++==--::··...
.··:::::::·····::-==++****+++=--:::·······:::----===---:::··... ...··:--=++*##%%%%@@@%%%%##***++===----::··...
.·:-+#%@@@@@@@#*+=-:··..··::-=+*#%@@@@@@%#*+=-··. ..·:-=*#%@@@@@@@#*+-:. .·-=*%@@@@@@@@@%*=-·..
.·:-=+*****+=-::·... ...·::-==++****++=--:··.. ...·::-=++*****++=-:·. .:-+#%@@@@@@@@%#+-:·.
..···::::···.. ...····:::::····.... ....···::::::::··.. .·=*#@@@@@@@@%#+-:·.
.·+*#@@@@@@%#*+=:·.
·:+*#%@@@%%#*+-:·.
.::+*#%%%##*+=-:·..
..:-**####*++=-:··.
..::-*****++==-::··..
.·:::--++++++==--::···...
....·················........ ......:-++++++++===---::::···...
........ .....··········... ..··::--====+++====---:::·····...······::::------****+++===-----:::::··...
...........··::--======--:··.. ..·:-==+*###%%###**+==-:::·······::--==++**########**+++===-----------:··..
.·-=*#%@@@@@@#*+=--:::::::-=++*#%%%%%%#*+=:·. .·:=+#%@@@@@@@@%#+=:·. .·:=+#%@@@@@@@@@%#+-:·..
..·:==+****++=--::·······::--===+++++==-::·.. .·::-=+******+=--:·. .·=*#@@@@@@@@@@#*=-:·..
..···:::····.............············.... ..···::::::···.. .·:*#@@@@@@@@@%#+=-:·...
·:*#%@@@@@@%#*++-::··..
·:+*##%%%##**+==-::··..
.·:=++****+++===--::··..
··-========------::··..
···:--------------::··..
....··:::::::::::-----::··.
.............. ....····:::::::::::····.... ··:::·········:::-----::·..
...··::::::::···... ..··:-==+**********++=--::··............··········:::········::::-----:··.
..··:--=======-::·.. ..·:-=*#%@@@@@@@@@@@%#*+=-::···..····::--===+++===---:::·····:::------::·..
.·:-=+*#%%#**+=-::··....··::-==+**####**+=-·. .·-+*%@@@@@@@@@@@%*+=:··......·:-=+#%%@@@@@%#*+=-:··....
.··:-==+===--::·····.·····::---=====--::·.. .·:-=+*********++=-:·.. ..··:-*#%@@@@@%%#*++=-::···..
...·······...............·········.... ..···::::::::···... ..:--#%%%%%%##**+++==--::·.
...--**####********++==-:·.
..::+++++++++++*****+=-:·.
..··------===++**##*+=-:·.
··::::::--==+**##*+=-:·.
......···::-=++***+=-:·.
.... ......·::-==++++=-:·..
...···::::::::···.... .....···········..... ..... ...··:--====-::·..
..·:--=++*****++=--:··.. ..··::-==+++***++==--:··... .....······.............··::::--:::·...
..·:-+*#%%@@@@%##*=-:··.. ..·:-=+*#@@@@@@@@@@%#*=-::·... ...··:::-----:::····....·····:::::···....
..·::-----::·.. ..·:-=+##%@@@@@%#*+=-·.. .·-=*#@@@@@@@@@@@%#+=-::·····::-==+*****++=--:::::::::···.
..········... ...··:--====+===--:··. .·:-=++*********+==-::········:::--**+++==-------====--:·.
...... ......········.... ..···:::::::::····..............---+===------=++****+=-:·.
...::=-------==+*#%%#*+=-:·.
..··::::::--=+*#@@%#*+=-:.
..·.·····:-=+*#%@@%#*+=-·.
. .....··:-=+*#%%@%#+=-:·.
... ...··:-==+***#*+=-:·..
. .. ...··::--=====--:··..
...··:::::::::::::··... ............. ........ . .......···:::::::::···....
..·:-=+************+=-:·.. ..·::--------::··.. ....·······..........··········............
.·-=+#%@@@@@@@@@@@@%*=-:·. ..·:-=+*###%##*++=-:·.. ....···::::···········:::::···.. .......
..·······.. .·:-+*%@@@@@@@@@@@@#*+-:··......··:-=+*##%%%%##*++=--:::::--===+++==--:··.. ...·:-====-:·.
....... ..··:-==+********++=-::·.........··::--===+++===---:::::::---------::::·......··:-+****+=-:·
....····::::::····.... .....··············.··············.:···......·:-=*#%%#*+=-:·
...·......··:=+*#%@%#*+=-:·
. ......·:-=+*#%@@%#*+=-:·
....··:--=+*##%%#*+=-:·.
..···::-==+++***++=--:·.
... ...··:::------====---::·.
..·.....::::::::::···:::::::··.
...···::::::::···.. ........... ............ .·:::::::::··.......····:···.
..·:-==+*********+=-:·. ..·::::::::···... ..................···-----::··.. ..······..
.·:-+*%@@@@@@@@@@@%#+-:·. .··:-==+++===-::···....................··::--=====-::·. .·····..
..·::--==--::··... ...··:-=*#%@@@@@@@@@@%#*+=-::·······::--------:::······::-=++*####*+=-:·. .·:-====--:·.
....····.... ...··::-==+********++==--::·········:::::::·····....···:--==+++++==-:·.. .·::-=+++==-:·.
....····::::::::·····....................... .....···:::::::··..... .··:-==+**++=--:·.
.. ......··::-==++**++==-:·.
. ..···:::---==++**+++=-:·.
..··:::--------==+++++=-:·.
...::----::::::--=+++++=-:·.
......:::---::·····::-==++==-:·
..::-:::=--:··....··:-=====-:·.
..··········... .............. .......... .......:-++==-::·......··:--=--::·.
.·:--=++++++=--:·.. ...···:::::··············...... ...··:::--+==-:··... ...··:---::·..
.·:=+*#%@@@@@%#*=-:·. ..··:::-------------:::··... ..·:-===+==--:··.......··::--:··.
·:-=+*##%#**+=--:········::--=++*******++==---::::----====--:··. .·:-=*#%@@@%#*+-:·. ..···::::··.
..··::----:::····....····:::----===-----::::::::::-------::··.. ..·:-=++***++=-::·.. . ...··:::---::·.
......... .......············..........··········... ...··:::::::···....·..·.........··:---===-:·.
...··::···....··:-==+++=-:·.
..::--::······:-=+***+=-:·.
··-:=-::·····:-=+###*+=-:·.
···-:--::···::-=+*%%#*+=-:·
······-::-::···::-=+*#%%#*+-:·.
··---::·::····::-=+*##%#*=-:·.
...········.... ............................ ........::--::·····:--=+**##*+=:·.
.··:--=====--:··... .......·········:::-----:::··... ...···::::···:······::-=++****+=-:.
.·:=++*****++=--:··.... .........···:--=+++++++=-:··. ..··::::::···....··:-==+*****+=:·.
.·:-=+**##*++=--:::·····::::::::::···........·:--=+*####*+=-·. .:-=+*####**++==-::··.. .··:::··.
.··::-----:::·········::::::::···.... ...·::-==++++=--:·. .··:--======----:::····. .··:----:·.
....................·····...... ...···:::::···.. ....··············.::···. .·-=====-:·.
··.:·:·.. ..·:=+***+=-:·.
·.:·:·.....·:-=+*%%#*+=-:.
·.:·:·····:-=+*%@@%#*+=-·.
·..··::::--=+#%@@@@%#*=-:..
··....···::-==*#%@@@@@@%*+-:·.
..···...::-=+*%@@@@@@@@#+-·..
....·········.............................·········..................... ....··:-=+#%@@@@@@@%*=:·.
..·:::-------:::::::::······..........··::---===---::················... ..·-=+#%@@@@@@%#+=:.
..·::--==============---::··.. .··:-=+++*+++=--::::::::::-:::··. .·:=+#%@@@@@@%*+-·.
██████╗ ██████╗ ██████╗ ██╗ ██╗███████╗███████╗ ██████╗ ██████╗ ███╗ ███╗██████╗
██╔══██╗██╔══██╗██╔═══██╗██║ ██║██╔════╝██╔════╝██╔════╝██╔═══██╗████╗ ████║██╔══██╗
██████╔╝██████╔╝██║ ██║██║ █╗ ██║███████╗█████╗ ██║ ██║ ██║██╔████╔██║██████╔╝
██╔══██╗██╔══██╗██║ ██║██║███╗██║╚════██║██╔══╝ ██║ ██║ ██║██║╚██╔╝██║██╔═══╝
██████╔╝██║ ██║╚██████╔╝╚███╔███╔╝███████║███████╗╚██████╗╚██████╔╝██║ ╚═╝ ██║██║
╚═════╝ ╚═╝ ╚═╝ ╚═════╝ ╚══╝╚══╝ ╚══════╝╚══════╝ ╚═════╝ ╚═════╝ ╚═╝ ╚═╝╚═╝ $ open /benchmarks/browsecomp▌highest recorded ██░░░░░░░░░░░░░░░░░░ 9.9 Accuracy (%) · o1-2024-12-17ready
BrowseComp
Agents / Search
BrowseComp evaluates web browsing and the ability to identify answers from multiple clues. This catalog supports introductions and discussion of search methods without republishing decrypted questions or answers.
browsecomp
History
OpenAI introduced BrowseComp in 2025 to test agents on difficult web searches whose answers must be identified from multiple clues. The benchmark targets the persistence and information gathering needed to locate obscure facts, rather than only the ability to answer familiar factual questions from memory.
At a glance
- License
- MIT (declared in the GitHub repository)
- Released
- Repository created on GitHub 2024-04
- Maintainer
- openai (GitHub organisation)
- Answer access
- This catalog has not attempted decryption and does not republish decrypted questions or answers.
- Highest recorded
- 9.9 Accuracy (%) — o1-2024-12-17 (paper)
What a problem looks like
Web-browsing questions that require identifying an answer from multiple clues.
Reported scores
Evaluation conditions differ between rows, so this is not a ranking. Rows marked vendor are reports by the model’s own organization.
| Model | Score | Metric | Setup | Reported by | Source |
|---|---|---|---|---|---|
| o1-2024-12-17OpenAI | 9.9 | Accuracy (%) | Without browsing; medium reasoning effort. | paper | Table 3; Section 4.1 model identifiers |
| gpt-4o-search-preview-2025-03-11OpenAI | 1.9 | Accuracy (%) | With browsing. | paper | Table 3; Section 4.1 model identifiers |
| gpt-4o-2024-08-06OpenAI | 0.6 | Accuracy (%) | Without browsing. | paper | Table 3; Section 4.1 model identifiers |
Why it is hard
Requires finding information on the web and combining multiple clues.
Limitations and criticism
A BrowseComp result reflects the browsing system, available tools and search budget in addition to the model. Results obtained without browsing therefore measure a different setup from results obtained with web access. Its focus on identifying answers from clues does not directly measure broader browser tasks such as completing transactions or editing documents.
How much room is left
The paper results recorded here report 9.9% accuracy for o1 without browsing at medium reasoning effort, 1.9% for GPT-4o search preview with browsing, and 0.6% for GPT-4o without browsing. These particular results are far from the ceiling and show considerable remaining difficulty for the evaluated systems. They are not a current leaderboard maximum, and their different browsing setups prevent treating them as a controlled comparison of model capability alone.
Reference links
- Repository
- Issues
- Site
Related benchmarks
Discussion
1 thread and 0 posts on this page. Anyone may reply; no account is required.
Contribute
Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.
Public problem summary
Catalogue metadata only; public problem statistics are not available.
- Public problems
- —
- Formats
- —
- Problems with images
- —
- Answer availability
- —
- Configs
- —
- Splits
- —
- Awaiting a first discussion
- —
- Problems with a discussion
- —
- Public contributions
- —