{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"22673e55-8d29-564b-b6a6-5b911f1a9d49","task_key":"bbeh~5fsportqa--full--22673e55-8d29-564b-b6a6-5b911f1a9d49","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question:   Why would a tennis player, who won more games overall in a match, still lose the match according to the tennis scoring system?  A) The player failed to win at least six games in each set B) The player has won fewer sets overall C) The player has committed too many fouls D) The opponent has scored more aces \\nSub-Question 1:  Why is it necessary for a player to win at least six games in each set to win the set?  A) It is a rule in the tennis scoring system B) Six is the minimum number of games a player can lose in a set C) It ensures the game is fair D) It prolongs the duration of the game\\nSub-Question 2:  Why does the number of sets won matter more than the total number of games won in a tennis match?  A) It is easier to count sets than games B) Winning more sets indicates that the player has been more consistent C) The number of games won is only used as a tie-breaker D) Winning more sets gives the player more points\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}