{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9d757db4-ad72-52ee-9925-9a315530bf27","task_key":"bbeh~5fsportqa--full--9d757db4-ad72-52ee-9925-9a315530bf27","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and a sub-question. Each question comes with…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and a sub-question. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: Why would a throw-in be awarded in a football match?   A) The ball has completely crossed the touchline, last touched by an opposing player. B) The ball has completely crossed the touchline, last touched by a player from the same team. C) A player has committed a foul. D) The ball has completely crossed the goal line, last touched by an opposing player.\\nSub-Question 1: Why would the ball crossing the touchline result in a throw-in?  A) The ball is considered out of play when it has crossed the touchline. B) The ball has crossed the touchline, but it was last touched by the goalkeeper. C) The ball has crossed the touchline, but it was a deliberate pass back to the goalkeeper. D) The ball is considered in play when it has crossed the touchline.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question is A and C and for the sub-question is D, your final answer must be \\\"AC, D\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}