{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"327549f7-02b5-594a-bf78-b9fdeac21b36","task_key":"bbeh~5fsportqa--full--327549f7-02b5-594a-bf78-b9fdeac21b36","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question:   What is the course of action taken by the referee in a football match when a penalty kick is about to be taken, but the weather conditions suddenly change, causing the pitch to become unplayable due to heavy rain, considering that the team awarded the penalty is leading by one goal and there are only five minutes left in the match?  A) The referee cancels the penalty and declares the match over due to weather conditions. B) The referee allows the penalty to be taken and then immediately ends the match. C) The referee suspends the match and resumes it later from the point of the penalty kick when conditions allow. D) The referee declares the match a draw due to unplayable conditions.\\nSub-Question 1:   What is the usual procedure when a football match is suspended due to weather conditions?  A) The match is resumed later from the point it was suspended. B) The match is restarted from the beginning. C) The match is declared a draw. D) The match is cancelled and not resumed.\\nSub-Question 2:   What happens to a penalty kick if a football match is suspended?  A) The penalty is cancelled. B) The penalty is taken immediately before the match is suspended. C) The penalty is retaken when the match resumes. D) The penalty is awarded as a goal.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}