{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3a4d45b8-090a-52b3-b6fe-6cf99962967e","task_key":"bbeh~5fsportqa--full--3a4d45b8-090a-52b3-b6fe-6cf99962967e","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question:  Why did the coach of Team A decide to substitute the goalkeeper in the 65th minute of a football match when the team was leading 2-0, the goalkeeper has not received a red card, and the goalkeeper's performance was not noticeably poor?  A) The coach wanted to give the substitute goalkeeper some game time. B) The coach received information that the goalkeeper was feeling unwell. C) The coach was implementing a pre-planned strategy. D) The coach was unsatisfied with the goalkeeper's performance.\\nSub-Question 1:  What event occurred just before the 65th minute of the match that might have influenced the coach's decision?  A) Team A scored another goal. B) The goalkeeper of Team A made a spectacular save. C) The coach of Team A received a message from the medical team. D) The striker of Team B was substituted.\\nSub-Question 2:  What was the previous record of the substitute goalkeeper when he was introduced in the middle of a match?  A) He has always maintained a clean sheet. B) He has conceded goals in most matches. C) He has an average performance record. D) He has never played in the middle of a match before.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}