{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"76461028-7d5a-522f-8f81-984b9f77b17e","task_key":"bbeh~5fsportqa--full--76461028-7d5a-522f-8f81-984b9f77b17e","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: Why would a referee award an indirect free-kick to Team A during a professional football match, despite Team B player's tackle on a Team A player inside Team B's penalty box being seemingly fair and the ball being in play at the time of tackle?  A) Because the referee thought it was a direct free-kick foul. B) Because the referee made a mistake. C) Because the Team B player, while tackling, was found guilty of dangerous play, which is sanctioned as an indirect free-kick foul. D) Because the Team B player was offside at the time of tackle.\\nSub-Question 1: Why is dangerous play considered an indirect free-kick foul, not a direct free-kick or penalty?  A) Because it doesn't involve physical contact. B) Because it is less severe than other fouls. C) Because it is not a foul at all. D) Because it involves the use of hands.\\nSub-Question 2: Why would a player be considered offside even when they are inside their own penalty box?  A) Because the player was closer to the opponent's goal line than both the ball and the last opponent when the ball was played. B) Because the player was not involved in active play. C) Because the player was in the opponent's half of the field. D) Because the player was behind the ball when it was played.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}