{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2c6a4194-aa60-5ef1-901d-85bbbaeb6987","task_key":"bbeh~5fsportqa--full--2c6a4194-aa60-5ef1-901d-85bbbaeb6987","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: In a football match, which procedures are typically followed when the game faces a penalty due to environmental or weather reasons, such as a storm or extreme heat?  A) The game continues with regular breaks for players to hydrate. B) The game is immediately suspended until further notice. C) The game is relocated to an indoor stadium nearby. D) The game is rescheduled to a later date.\\nSub-Question 1: If the game is suspended due to environmental or weather reasons, which are the typical next steps taken by the organizing committee?  A) The committee waits for the weather to improve and then resumes the game. B) The committee reschedules the game to a later date. C) The committee relocates the game to an indoor stadium if available. D) The committee cancels the game and refunds all ticket holders.\\nSub-Question 2: If the game is relocated to an indoor stadium nearby due to environmental or weather reasons, which are the typical considerations taken into account by the organizing committee?  A) The availability of the indoor stadium. B) The capacity of the indoor stadium to accommodate spectators. C) The distance of the indoor stadium from the original location. D) The rules and regulations of the indoor stadium.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}