{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"849336e7-d6e8-5f3c-a36c-f6d2c144699e","task_key":"bbeh~5fsportqa--full--849336e7-d6e8-5f3c-a36c-f6d2c144699e","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: In a professional tennis match, under what circumstances would a player be granted a second service opportunity after committing a fault, considering the different scenarios of service rules?  A) When the player's first service hits the net, posts or singles stick and then lands in the correct service box. B) When the player's first service hits the net, posts or singles stick and then lands outside the correct service box. C) When the player's first service directly lands outside the correct service box. D) When the player's first service is interrupted by external factors.\\nSub-Question 1: What is considered a fault in professional tennis service rules?  A) When the player's service hits the net and lands in the correct service box. B) When the player's service hits the net and lands outside the service box. C) When the player's service directly lands outside the service box. D) When the player's service is interrupted by external factors.\\nSub-Question 2: Under what conditions is a player granted a second service opportunity in professional tennis?  A) When the player's first service hits the net, posts or singles stick and then lands in the correct service box. B) When the player's first service hits the net, posts or singles stick and then lands outside the correct service box. C) When the player's first service directly lands outside the correct service box. D) When the player's first service is interrupted by external factors.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}