{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fbf350cc-6f31-5679-8630-861418179bc1","task_key":"bbeh~5fsportqa--full--fbf350cc-6f31-5679-8630-861418179bc1","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and a sub-question. Each question comes with…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and a sub-question. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question:  Why might a football player be deemed offside during a match?  A) The player is nearer to the opponent's goal line than both the ball and the second last opponent when the ball is played. B) The player is involved in active play by interfering with an opponent or gaining an advantage from being in that position. C) The player is standing in his own half of the field. D) The player is behind the ball when it is played.\\nSub-Question 1: Why might a player not be penalized for being in an offside position? A) The player receives the ball directly from a goal kick. B) The player is in his own half of the field when the ball is played. C) The player is not involved in active play as determined by the referee. D) The player receives the ball directly from a throw-in.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question is A and C and for the sub-question is D, your final answer must be \\\"AC, D\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}