benchmarks.wiki / Public workspace

  .·::--=+********++===--------::··..      ..·:=+**#####*+==-::······:::------------------::::······::--==++**++=-:··.  
  ...···::---------:::··········...         ..·::--=====---::························::::::···:::::::--==+**##*+=-::·.  
      ...................                     ....·········..........            ...........::--::::--==+**##%#*+=-:·.  
                                                                                          ··--==------==+*##%%#*+=-:·.  
                                                                                          ··=-+==-----==+*#%%%#*+=-:·.  
                                                                                          :·==*+=------=+*#%%@%#+=-:·.  
                                                                                          ::+=*+=--::--=+*##%%%#+=-:·.  
                                                                                          ::+=*+=-::::--=+*#%%%*+=-:·.  
                                                                                          ::+=*+=-:::::-=+*##%#*+=-·..  
                                   ..··:::::::::··.             ..··::::::··.........·······==*=-::···::-=+*#%#*+=:·..  
   ...           .....          .··:-+*********=-·.           .·-=********+=-:··..···:--==+===+=-:·····:-=+*###*+-:·.   
  .....         ...·······...··::-+*#@@@@@@@@@#+-·.          ·-+#@@@@@@@@@%*=-:··.··:-=+*####*+=-:··..··:-=+*#*+=-:·.   
  .·:-=++*##**++=======++++++==-:·... ...·:=+#%@@@@@@%*+=:·..     ..·::-==++++++++++===--:::·················:·····...  
   .··::--===---:::::::::---:::··...   ...·:--=+++++==-::·...    ....···:::::::::::::::·····..·······:::::::::::····..  
     ....······.................          ....········.....          .......................·.······:::-------:::···..  
                                                                                          ..·······:::---=====-:::··..  
                                                                                          ..··::··:::--========-::··..  
                                                                                          ..:·:::::::--==+++++=--:··..  
                                                                                          ·.::-:::::--==+++*+++=-::·..  
                                                                                          ··::-:::::--=++*****+=-::·..  
                                                                                          ··-:-:::::--=++*****+=--:·..  
                                ..···::::::::··.             ..··:::::::···........·········-:--::::--=++*####*+=-:·..  
            ......         ...·::-=+*******+=-·.            ·:-+********+=-::····::--==++==--:-:::::--=++*####*+=-:·..  
              .......·····:::-=+*#%@@@@@@@#*=-·.          .·=*%@@@@@@@@@@#+=-::::--=+*#%%##*+=-::::::-==+*####*+=-:·.   
  .··:------:····::--=++*++==-:·.....·:-=*#@@@@@@@@#+=:.         .·:-=+*#%%%%%##**+==--:::······.....        ..·····..  
   ..··::::····.····::------::··......·::-=+++*++==-:·..        ..·::--========---::···..............        ...···...  
     ..................·····.....    .....········...            .....········........        .........     ...····..   
                                                                                              .................····..   
                                                                                              ................·····...  
                                                                                              .....·····....········..  
                                                                                            ......··················..  
                                                                                            .....····::::::::::::···..  
                                                                                            ....····::::::-----:::··..  
       .......              ....···::::::··..              ..··::::::···................... ..·····::------===---::·..  
    ......·...............··::--==++**++=--:·.           .·-=+*******+=-::······:::----::···..····::---====+++==-::·..  
   .......··..·······:::--==++**##%%###*+=-:.           ·:=*%@@@@@@@@@%*+--::::--=++++++=--::····:::-==++++***+=-::·..  
   ......      ..·:-=++++=--:·....··:=+#@@@@@@@@#*=:.          ·:=+#%@@@@@@@@%#*++=---::::::::··..         .·:----:·..  
    .....      ..··::------::···.···::-==+++++=-::·.         ..·:-=++******++=--::···.........·...         .·::---:·..  
                 ....·······............······...            ...·····::::·····.....         .....          ..·::-::·.   
                                                                                              ...           .·:::::·.   
                                                                                              ..            .·::::··.   
                                                                                                            .··:::··.   
                                                                                                           ..··:::··..  
                                                                                                           .··::::··..  
                                                                                                 ...    ...··::::::··.  
     ..........          ......·········...             ..····::····....                       ..........···:-----:··.  
   ..·········.........···:::---=====--::·..          .·:-=++***++=--:··..............        .....·····:::--===--::·.  
  ..····:::::····::::---====+++++++++==-::·.         .·-=*%@@@@@@@%*+=-:········:::···...    ....···:::---==+++==-::·.  
              .·:==++++==-:··...·:-=+#%@@@@@@#*=:.          .:-+#@@@@@@@@@@@#*++==---====+==--:·.       .:-+*###*=-:..  
              ..·::------::·······::--======-:··.          .·:-=+*********++=-::········::::····.       .·-=*##*+=-:.   
                ...········...................            ...···::::::::::···....      .....···..       .·:=+***+=-·.   
                                                                                            ..·.        .·:-+***+=:·.   
                                                                                            ....         .·-=+++=-:·.   
                                                                                            ....         .·:-=+==-:·.   
                                                                                            . .          .·:-====-:·.   
                                                                                              .          .·:-===--:·..  
                                                                                                         .·:-====-:··.  
    ............        ..................          ...·········.....                                  ..·::-=+==--:·.  
  ..···::::····.......·····::::------:::··...     ..··:--==+===--:···......                         ...··:-==++++=-:·.  
  .··:::::::::·::::::-------==========--::·...  ...·:-=+*#%%%%#*+=-:··.......                ......···::--=+****+=-:·.  
          .·:-=+*****+=-:··....·:-=+#%%@@%#*+-:.         .·:=*%@@@@@@@@@@%*+==---==++****+=-:·.      .·:=*#@@@%#*=:·.   
           ..·::--===--::····.····::------::·..         .··:-+**********+=--::····:::-----::·..      .·-=*%@@@%#*=-·.   
             ...·········..................             ...··:::::::::::···............·....·..      .·-=*%@@@%#*=-·.   
                                                                                          ..·..      .·-=*%@@@%#*=-·.   
                                                                                          ..·..      .·:=+#%@@%#*=:·.   
                                                                                          .....      .·:-+*%@@%#+=:·.   
                                                                                          .....      ..:-=*#%%#*+-:·.   
                                                                                            ...       .·:=+*###*=-:·.   
                                                                                            ...      ..·:-=+*#*+=-:·.   
    .........            ..........................············..........                   ...      ..·:-=+*#*++-:·..  
  ..·······............····::::------::::········:::--======---::::·····...              ............··:-=+**##*+=-:·.  
  .··:::···········:::::----===++++++===-:::···::--=++**##**++==--::::···..             ..........···:--=+*####*+-:·..  
      ..·:=+*#####*+=-:·..  ..·:-=+*#%%%#*+=:·.        .·:=*#@@@@@@@@%*+-::···::-=+++++=-·.         .:-+*%@@@@%*+-:·..  
        ..·::--====--::··......···::----:::··..       ..·:-=+********+=-:··...···::----:··.         ·:=*%@@@@@%#+=:·..  
           ...········.....        .........          ...···:::::::::···...    .....··....         .·-+#@@@@@@%#*=:·.   
                                                                                                   .:=*%@@@@@@%#*=:·.   
                                                                                                  .·:=*%@@@@@@%#+-·.    
                                                                                          . .    ..·-=*%@@@@@@%#=:·.    
                                                                                          . ...  ..·:=*#@@@@@@%*=:.     
                                                                                          . .......·:-+*#%@@@@#+-:.     
                                                                                          .........·:-=+*#%%%#*=-:·.    
     ..                 ....................··························...           ..........·...··::-=+*####*=-:·..   
  ........       ....····:::-----------::::::----==================--::··.        ...·······..······:--=+***#*+=-:·..   
  .......   ....··:::---===++********++========++**#####**********++==-:·.         .··:::::::::··:::--=++*****+=-:·..   
   ..·:-=*#%%%##+=-:·.      .·:-=*#%%%%%#*=-:..       .:-=*#%@@@%#*=-·..    .··:----:·.            ·:=+*#%@@%%*+=-:·..  
     ..··:-------::··...   ...··::-------::·..        .·:-==+**++=-:··.      ..··::··..           .:-+#@@@@@@@%*+-:·.   
         ....·····......        ............           ..···:::::··...         .......            ·-+#@@@@@@@@%#+-·.    
                                                                                                 .:=*@@@@@@@@@%*=:.     
                                                                                                .:-+%@@@@@@@@@%+-·      
                                                                                              ..·:=*%@@@@@@@@@*=·.      
                                                                                            ...·:-+*%@@@@@@@@#+:.       
                                                                                          . ..·::-=*#%@@@@@%#+-·.       
                                                                                          ..··:::-=++*##%##*+=-·..      
                   ...................·····················::::::···...         ....····....··::::--=+++**++==-:·..     
               ...···::::-----------------==========---====+++++++=-:··.      ..··::-----:::··:::::---===++===-:··..    
            ..·::--===+++***********+++++****######******##%%%@@%##+=-:·.     ..·:-=++++++==---::::----=======-:··..    
  .··:-+*###*+=-:·.       .·:-+*%@@@@@@%*+-:..      .·:=+*####*=-:.       ..·:::::·.             .:-+*#%%%%%##+=-:·..   
    ..··:::::::··....    ..··::-==++++==-:·..        .·:--====-::·.         ..····..            .·-+#%@@@@@@@%*=-·..    
         .............      .....······....           ...······...             ..               ·-+#@@@@@@@@@@*=:.      
                                                                                               ·:+#@@@@@@@@@@%+:.       
                                                                                              ·:=*@@@@@@@@@@@*-·        
                                                                                             .:=*%@@@@@@@@@@*=·         
                                                                                            .·-+*%@@@@@@@@%*=:.         
                                                                                          ..·:=+*#%@@@@@%#+=:.          
                                                                                          ..::==++******+=-:·..         
              .........................·····............····:····..           ..··········..::-------=----:::··....     
           ...··::::-----::::::::::::---------:::····::--=+++++=-:··.       .·::-==++++==--::·:::::::::::::::::···...   
         ..·::-==++++****+++++=====++++******++==----==+*#%%@@%#*+-:·.     .·:-=*#%%@%%##*+=-::········::::---:::··...  
  .·::--===-:·..       ..:-=*%@@@@@@@@%*+-·.     ..·-=+*###*+=-:·.    ..·:--==--:·.           .·-=+*#%%%%##*+=-:·..     
    .....····...........··:--=+******+=-:·.        .·:--===--:·..       ..···:···.            ·:=+#%@@@@@@%#*=:·.       
           ..................··········..            ..·····...            .....              :-+#@@@@@@@@@#+-·.        
                                                                                             .-+#@@@@@@@@@@*=:.         
                                                                                            .·+#@@@@@@@@@@#=:.          
                                                                                            ·:*%@@@@@@@@@#=:.           
                                                                                          ..:-#%@@@@@@@%*=:.            
                                                                                          ..--*##%%%##*=-:.             
                                                                                          ..::=======--:··..            
          ....·····.....         .........           ...·····..             ..··::::::····..··::···················...  
        ..·:::-------::···......····::::···..      ..·::-===--:·.         .·:-=+******+=-::·..... ....···::-----::··..  
       .·:-==++****+++=--:::···:::--====--:··..   .··-=+*###*+=-:·.     ..:-+*%@@@@@@@%#+=:·..      ..·::-===+==--:··.  
  ...······...     ..·:-=*#@@@@@@@@@@#+=:·.   .·:-=*#%%%%#*+=-::···:--=++***++=-:··..    ..·:-=+*##%%##*+=-:··.         
        ......········::--=+********+=-·..      .·:-==++==--:··.....··:::----::·..         .··=*#%@@@@%#*+-:·.          
          ...............····::::::···..          ..·······...        ..........            ·:+#@@@@@@@%#+-·.           
                                                                                            :-#@@@@@@@@%*=:.            
                                                                                           .:=%@@@@@@@@#+:.             
                                                                                          ..-=@@@@@@@%#+-·              
                                                                                          ..-=%%@@%%#+=:·.              
                                                                                          ..--+++++=--:·.          ...  
                                                                                          ..··::::······.........···..  
       ..·········...           .....               ....·....            ..···:::::::··...  .     ...···::::---::::··.  
     ..·:-=======-::·..       ...·····..           .·::---::·..        ..·-=+*******+=-:·.         ..·:--=+++*+==-::·.  
    ..·:-=**###**+=-::·..    ..··:::::·.          .·:-++**+=-:·.      .·:=*%@@@@@@@@%*=:·.         .·:-=+*#####*+=-:·.  
     ....······.···::-=*#@@@@@@@@@%#+-:·.....·:-+*#####*++=------==++**##***++=--::::::::--=++******+=-:·..      .....  
       ..····:·····:::-==+********+=-··.    ..·::-======--::·····:::--====---::···.......··::-*#####*+=-:·.       ....  
        ..·······.....····::::::::··..         ..········............·······....          ..--#%%@@%#*+-:.         ...  
                                                                                          ..-=%@@@@@%*=-·.          ..  
                                                                                          ..==@@@@@%#+=:.          ...  
                                                                                          .·==%@@%%#+=:·.          ...  
                                                                                          ..--****+=-:·.         ...·.. 
                                                                                          ..··-----::··...     ..····.. 
                                                                                            ....················::::··. 
     ..···:::::··...        ........             ...····...            ..··:::::::···..          ..·::--======----::··. 
   ..·:-=+***++=-:·..      ..··:::··.           .··:-----:··.        .·:-=+*******+=:·.          .·:-=+***#***+==--:··. 
   .·:-+*#@@@%#*+-:·.      .··:---::·.          .·-=+***+=-:·.      .·-=*%@@@@@@@@#+-·          .·:-+*#%@@@@%%#+=--:·.. 
    ..··:::::::·:::-=+*#%@@@@@@#*=-·.      .··:-------:::::::---==++*******++++============++++===-:··.      ..··:::··. 
     ..··::::::···:::-==+*****+=-:·..      ..··::::::::·······:::---======----:::········::::-+++==-::·..    ..··:::··. 
      ...······......····::::::··..          ....................···········.....       ....--****+=-:·..     ..··::··. 
                                                                                          ..-=####*+=-:·.      ..·····. 
                                                                                          .·==%%%#*+=-·.       ..·····. 
                                                                                          ..-=###**=-:·.       ..·····. 
                                                                                          ..:-++++==-:·..     ..··::··. 
                                                                                           .··-------::··......··:::··. 
                                                                                             .··::-----:::·····::::::·. 
    ...···::::··...    ....·····...           ..········...          ...··::::::··..          .·::-==++===-----::::::·. 
  ..·:-==+****=-:··.....··:-----::·.         .·:-===+==-:··.       ..·:-+******+=-:·.         .·:-=+*###**++===---:::·. 
  .·-=+*#%@@@%#+-:·....··:-=++++=-:·.        .:-=*#####*=-:·.     .·:=+#%@@@@@@#*=:.          .·-=*#%@@@%%#**+==--::··. 
   .··:-----::::::-==+*###**+=:·.             ..........··::---===+++++++***++++++=====-------:::··...   ..·:-=+++=-:·. 
   ..··:::::·······::--====--:·..             ...........···:::----=======---:::············::---::···....··:-=+++=-:·. 
     ...···...........·······..                        .......·············....          ...::+++==-::··..··:--====-:·. 
                                                                                          ..--****+=-:··...··:--==--:·. 
                                                                                          .·==###*+=-:··.....·::----:·. 
                                                                                          .·==###*+=-:·... ..··::--::·. 
                                                                                          ..--****+=-:·... ...·::--:··. 
                                                                                          ..::+++++=-::·.....··:::::··. 
                                                                                            ··==+++==-::·....··:::::··. 
    ...···::··.... .....······...          ..··::::::···..          ...···:::···..          .·-=+++++=--:······:::::··. 
  .··:-==+**+=::··..··::--==--::·.       ..·:-=+****++=:·..       ..·:=++****+=-:·.         .·-=+*****+=-::··:::::::··. 
 .·:-=+*#%@@%*=-:····:-=+****+=-:·.      .·:=+#%@@@@%#*=-·.      .·:=+#%@@@@@#*=:·.        .·:-+*#%%##*+=--:::::::::··. 
  .·::-----::::::-===++==-:··.                     ..··::---=========+++++++++===-----::::::::····....···:-=*###*+=-:·. 
   .········....···::::::··..                       ..···:::---------------:::···......·······--::::···::-=+*#%#*+=-:·. 
    ........     ..........                          ......·············.....           ....:-++==--:::::-==*###*+=-:·. 
                                                                                          ··==##*++=-:::::-=+****+=-:·  
                                                                                          ··++%%#*+=-::··::-=+++++=-:.  
                                                                                          ··++@%#*+=-:·····:-==++==-·.  
                                                                                          ··++%%#*+=-:·....·:--===-:·.  
                                                                                          ··==###*+=:··.....·:--=--:·.  
                                                                                          ..-=##*++=:··.....·::----:·.  
    ...····..      ..........           ..··:::::::::·..           ...········...         ..--***++=-:·.. ...·::--::·.  
  .··:-===-:··.....··:::::::··.       ..·:=+*******+=-:·.         .·-=++***+==:··.       .··:-****+=-:·..  ..·::--::·.  
 .·:-=+*##*=-:··.··::-=++++=-:··..  ..·:-+*%@@@@@@@%*=:·         ·:=*%@@@@@@#*=:·.       .·:=+*****+=:··.....··:::::·.  
  .·:::-------===+++===-::··......              .·:--==++++====-------======------::::::::::::·······::--=+*#%%#*+=-:.  
  ...··········:::::::···...                    ..··:::---------:::::::::·····.......·····:::·--::::::--=+*#%@%#*+=-:.  
      .       ..........                          ....·············......            .......--+==------==+*#%@%#*+=-:.  
                                                                                          ··==**+==-----=+*#%@%#*+=-·.  
                                                                                          ::++%#*+=------=+*#%%#*+=-·.  
                                                                                          ::*+%#*+=--:::--=+*###*+=:·.  
                                                                                          ::*+%#*+=-::··::-=*##**+-:·.  
                                                                                          ::++%#*+-::····::=+***+=-:·.  
                                                                                          ··++#*+=-:··...·:-=++++=-:·.  
    ....                              ..·::::::::::·..            ..···:::···....    ....···==#*+=-:·.....·:-=++==-:..  
  .······..      ...····...        ..·:=+*********=-·.          .·:=+******+=-:·... ...·::--==**+=:··.  ..··-====-:·..  
 .·::--::·..    ..··:::::::··.....·:-=+#@@@@@@@@@#+-·.         .:=*%@@@@@@@%*=-·..   ..·:-++***++=:·..   ..·:-===-:·.



    ██████╗ ███████╗███╗   ██╗ ██████╗██╗  ██╗███╗   ███╗ █████╗ ██████╗ ██╗  ██╗███████╗
    ██╔══██╗██╔════╝████╗  ██║██╔════╝██║  ██║████╗ ████║██╔══██╗██╔══██╗██║ ██╔╝██╔════╝
    ██████╔╝█████╗  ██╔██╗ ██║██║     ███████║██╔████╔██║███████║██████╔╝█████╔╝ ███████╗
    ██╔══██╗██╔══╝  ██║╚██╗██║██║     ██╔══██║██║╚██╔╝██║██╔══██║██╔══██╗██╔═██╗ ╚════██║
    ██████╔╝███████╗██║ ╚████║╚██████╗██║  ██║██║ ╚═╝ ██║██║  ██║██║  ██║██║  ██╗███████║
    ╚═════╝ ╚══════╝╚═╝  ╚═══╝ ╚═════╝╚═╝  ╚═╝╚═╝     ╚═╝╚═╝  ╚═╝╚═╝  ╚═╝╚═╝  ╚═╝╚══════╝

Pause animation

Benchmarks

Compare the whole catalogue · Browse model reports

Choose a field to explore public problems and discussions.

Fields and counts refer to this catalog page.

Mathematics
AIME 2024
aime-2024
The 30 problems of the 2024 American Invitational Mathematics Examination (AIME I and II). Every answer is an integer from 0 to 999, with no options and no partial credit. It is a frequent reference point for reasoning m…
Full text30 tasks
Size
All 30 problems imported from the default / train selection, with a source-row limit of 200.
License
mit (declared on the Hugging Face dataset card)
Reference links
2
Mathematics
AIME 2025 I
aime-2025
The 15 problems of AIME I 2025. As one of the newest competition sets available, it leaves the least room for training-data contamination, and reading it beside the 2024 set makes that effect visible.
Full text15 tasks
Size
All 15 problems imported from the AIME2025-I / test selection, with a source-row limit of 200.
License
mit (declared on the Hugging Face dataset card)
Reference links
2
Mathematics
AMC (AIMO validation set)
amc
83 AMC 12 problems assembled by Project Numina as a validation set for the AIMO competition. They sit a step below AIME in difficulty, which makes them a useful lower rung on the same scale.
Full text83 tasks
Size
All 83 problems imported from the default / train selection, with a source-row limit of 200.
License
apache-2.0 (declared on the Hugging Face dataset card)
Reference links
2
Mathematics
FrontierMath
frontiermath
FrontierMath evaluates advanced mathematical reasoning using problems written by experts. This catalog links to the official introduction and public examples while excluding private evaluation problems.
Reference links
1
Mathematics
GSM8K
gsm8k
GSM8K is a set of grade-school word problems that each need several arithmetic steps. It remains a standard reference point for multi-step reasoning.
Full text1,319 tasks
Size
1,319 problems imported from the main / test selection (the full test split).
License
mit (declared on the source)
Reference links
3
Paper: arXiv:2110.14168
Mathematics
MATH
math
MATH organises 12,500 high-school competition problems into seven subjects and five difficulty levels, each with a full worked solution. It underlies much of the later mathematics evaluation work; this catalogue imports …
Full text5,000 tasks
Metric
accuracy on the final boxed answer
Size
5,000 problems imported from the test split across all seven subjects.
License
mit (declared on the Hugging Face dataset card)
Reference links
4
Repository: Official repository
Mathematics / Multimodal
MathVista
mathvista
MathVista gathers 6,141 mathematical problems that come with figures, charts or geometric diagrams, of which 1,000 form the public testmini split. It measures whether visual information can be carried into mathematical r…
Metric
accuracy over testmini and the full test set
Size
6,141 problems in the source, 1,000 in testmini; none imported.
License
cc-by-sa-4.0 (declared on the Hugging Face dataset card)
Reference links
4
Paper: arXiv:2310.02255
Mathematics / Multilingual
MGSM
mgsm
MGSM is a human translation of 250 GSM8K problems into eleven languages, measuring whether reasoning survives the change of language. The gap against English is the multilingual signal.
Full text2,750 tasks
Size
2,750 problems imported from the original TSVs: the same 250 grade-school problems in each of the eleven languages, kept as per-language pages.
License
cc-by-4.0 (declared in the original repository mgsm/LICENSE)
Reference links
4
Repository: Official repository
Mathematics / Science / Multimodal
OlympiadBench
olympiadbench
OlympiadBench evaluates scientific reasoning on Olympiad-level mathematics and physics problems. Its official description lists 8,476 English and Chinese problems with separate text-only and multimodal settings.
Full text200 tasks
Size
8,476 English and Chinese problems described across settings in the catalog introduction; this OE_TO_maths_en_COMP / train selection expects 200 task records with a source-row limit of 200.
License
apache-2.0 (declared on the Hugging Face dataset card)
Reference links
4
Repository: Official repository
Mathematics
Omni-MATH
omni-math
Omni-MATH evaluates mathematical reasoning on Olympiad-level problems. Its official dataset contains 4,428 problems accompanied by domain and difficulty information.
Full text4,406 tasks
Size
4,406 unique problems imported from the test selection of 4,428 rows; 22 rows repeat an earlier problem and are recorded as variants.
License
apache-2.0 (declared on the Hugging Face dataset card)
Reference links
3
Paper: arXiv:2410.07985