{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"503af70a-dc0f-5c5a-b1d7-ef7f42f8633a","task_key":"bbeh~5fsportqa--full--503af70a-dc0f-5c5a-b1d7-ef7f42f8633a","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: Under what circumstances in a football match is a penalty awarded?   A) When a player handles the ball deliberately in his own penalty area. B) When a player tackles an opponent in a manner considered by the referee to be careless, reckless, or using excessive force. C) When a player scores a goal directly from a throw-in. D) When a player is offside.\\nSub-Question 1: What constitutes a player handling the ball deliberately in his own penalty area?  A) When the player's arm or hand moves towards the ball. B) When the ball hits the player's hand or arm which is in a \\\"natural\\\" position. C) When the player's hand or arm is making the body unnaturally bigger. D) When the ball hits the player's hand or arm which is close to the body.\\nSub-Question 2: What is considered as a player tackling an opponent in a manner considered by the referee to be careless, reckless, or using excessive force?  A) When the player does not show due care for the opponent's safety. B) When the player makes contact with the ball before making contact with the opponent. C) When the player challenges for the ball without any chance of playing the ball. D) When the player makes a fair challenge with the opponent for the ball.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}