{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"086760df-d51d-5c72-b304-51cd317e0fff","task_key":"bbeh~5fsportqa--full--086760df-d51d-5c72-b304-51cd317e0fff","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: What are the correct procedures during a tennis service in accordance with the official rules, in the scenario where the server's first service attempt hits the net but lands in the correct box and the second service attempt is a fault?  A) The server gets another chance to serve because the first service was a let. B) The server loses the point because the second service was a fault. C) The server has to re-serve without any penalties. D) The server gets to serve again only if the first service was a let and the second service was not a fault.\\nSub-Question 1: What are the correct procedures when the first service hits the net but lands in the correct box?  A) The server gets another chance to serve. B) The server loses the point. C) The server has to re-serve without any penalties. D) The server gets to serve again only if the second service is not a fault.\\nSub-Question 2: What are the correct procedures when the second service attempt is a fault?  A) The server gets another chance to serve. B) The server loses the point. C) The server has to re-serve without any penalties. D) The server gets to serve again only if the first service was a let.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}