{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3f8db7ca-8ddc-5deb-b383-651a2b143e3e","task_key":"bbeh~5fsportqa--full--3f8db7ca-8ddc-5deb-b383-651a2b143e3e","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question:  How is a goal in football considered valid in terms of player's position and ball crossing the line, given the following scenarios? A) The ball completely crosses the goal line between the goalposts and under the crossbar, and the player who last touched the ball was not in an offside position when the ball was played. B) The ball crosses the goal line partially, but the player who last touched the ball was not in an offside position. C) The ball completely crosses the goal line between the goalposts and under the crossbar, but the player who last touched the ball was in an offside position. D) The ball completely crosses the goal line between the goalposts and under the crossbar, and the player who last touched the ball was not in an offside position when the ball was played, but the ball was handled by the player.\\nSub-Question 1: Considering the offside rule in football, when can a player be considered as not being in an offside position during play? A) When the player is nearer to the opponent's goal line than both the ball and the second last opponent when the ball is played. B) When the player is in his own half of the field of play. C) When the player is level with the second last opponent. D) When the player is level with the last two opponents.\\nSub-Question 2: When is the ball considered to have crossed the goal line in football? A) When the whole ball passes over the line, between the goalposts and under the crossbar. B) When any part of the ball passes over the line. C) When the ball touches the goal line. D) When the ball is within the goal area.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}