{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9343b2a6-bded-5979-b73c-d4b3634b750f","task_key":"bbeh~5fsportqa--full--9343b2a6-bded-5979-b73c-d4b3634b750f","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and a sub-question. Each question comes with…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and a sub-question. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question:   Why might a tennis player prefer to serve first in a match, despite the fact that serving first does not necessarily guarantee a win?  A) Serving first allows them to control the pace of the game. B) Serving first guarantees them the first point of the match. C) Serving first gives them a psychological advantage. D) Serving first allows them to win the match easily. \\nSub-Question 1:   How can serving first in a tennis match give a psychological advantage?  A) It can make the opponent feel under pressure from the start. B) It can make the opponent feel more confident. C) It can make the opponent feel tired. D) It can make the opponent feel relaxed.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question is A and C and for the sub-question is D, your final answer must be \\\"AC, D\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}