{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"dfaa8328-97bd-5a65-a645-736e6410e5b4","task_key":"bbeh~5fsportqa--full--dfaa8328-97bd-5a65-a645-736e6410e5b4","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: How might the roles of various officials in a table tennis match impact the final decision in a controversial scenario where a player's serve is deemed illegal due to the ball not being tossed high enough and the opponent claims that the player's free hand was not flat during the serve?  A) The umpire decides that the serve is illegal based on the toss height alone. B) The assistant umpire decides that the serve is illegal due to the player's free hand not being flat. C) The umpire decide that the serve is illegal based on both factors. D) The umpire overrules the assistant umpire's decision, deeming the serve legal.\\nSub-Question 1: In what scenarios might the umpire and assistant umpire's decisions conflict during a table tennis match?  A) The umpire believes the player's free hand was flat during the serve, but the assistant umpire disputes this. B) The assistant umpire believes the ball was tossed high enough, but the umpire disputes this. C) Both officials agree on the toss height, but disagree on whether the player's free hand was flat. D) Both officials agree on the legality of the player's free hand, but disagree on the toss height.\\nSub-Question 2: How might the umpire's and assistant umpire's role influence the final decision in controversial scenarios in a table tennis match?  A) The umpire can overrule the decisions of the assistant umpire. B) The umpire can not overrule the decisions made by the assistant umpire. C) The assistant can make a decision independently of the umpire. D) Both the umpire's and assistant umpire's decisions are final.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}