{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1c62d7f0-2413-52b7-9a78-a162b0cbc880","task_key":"bbeh~5fsportqa--full--1c62d7f0-2413-52b7-9a78-a162b0cbc880","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question:  In a doubles table tennis match, Team A consists of Player 1 and Player 2, and Team B consists of Player 3 and Player 4. If Player 1 served to Player 3 and the ball hit the net but still landed in the correct service box, and Player 3 failed to return it, what could be the possible outcomes? A) Point for Team A B) Point for Team B C) Let is called  D) The serve is repeated\\nSub-Question 1:  If a ball hits the net during service but still lands in the correct service box, what is the usual outcome? A) Point for the serving team B) Point for the receiving team C) Let is called  D) The serve is repeated\\nSub-Question 2:  If Player 3 fails to return a valid serve, what is the usual outcome? A) Point for Team A B) Point for Team B C) Let is called  D) The serve is repeated\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}