{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"972da74c-cece-5ac4-a4d4-3b1799319894","task_key":"bbeh~5fsportqa--full--972da74c-cece-5ac4-a4d4-3b1799319894","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question:  What was the resolution method used by the mediator in the conflict between Team A and Team B after Team A's coach was red-carded for offensive behavior towards the referee, considering that the referee had previously officiated a match where Team B's player was controversially sent off?  A) The mediator proposed a rematch B) The mediator recommended disciplinary action against the coach C) The mediator suggested a neutral referee for future matches D) The mediator decided to disqualify Team A\\nSub-Question 1:  What was the reason for the red card given to Team A's coach?  A) Offensive behavior towards the referee B) Violating the sideline rules C) Instructing the players to play unfairly D) Disrespecting the opposing team's coach\\nSub-Question 2:  In the previous match officiated by the referee, what was the controversy surrounding the sending off of Team B's player?  A) The player was wrongly accused of a foul B) The player was sent off without any valid reason C) The player received a red card instead of a yellow card D) The player was sent off for an action that usually doesn't warrant a red card\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}