{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d52d38d1-f9f0-529f-8ba6-b9dc5589ffcf","task_key":"bbeh~5fsportqa--full--d52d38d1-f9f0-529f-8ba6-b9dc5589ffcf","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: What is the consequence of a table tennis player who, after receiving a yellow card for unsportsmanlike conduct in a match for first time, subsequently disrespects an umpire's decision according to the International Table Tennis Federation (ITTF) rules?  A) The player will be disqualified immediately. B) The player will be given a warning. C) The player will be given a yellow card and red card at the same time. D) The player will be banned from future tournaments.\\nSub-Question 1: What does a yellow card signify in a table tennis match according to the ITTF rules?  A) A warning for unsportsmanlike conduct. B) Immediate disqualification. C) Respect for the umpire's decision. D) A player's exceptional performance.\\nSub-Question 2: According to ITTF rules, how should a player react to an umpire's decision?  A) A player can disrespect the decision if they disagree. B) A player should argue the decision if the player lose a point. C) A player can challenge the umpire's decision during the match but should respect the umpire. D) A player has the right to ignore the decision.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}