{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5a3aab7b-6867-5789-8a4b-f9278f3ed0d7","task_key":"bbeh~5fsportqa--full--5a3aab7b-6867-5789-8a4b-f9278f3ed0d7","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question:   Considering the different basketball leagues and the varying rules they follow, how many players are usually present in a team at the start of a match and how many substitutes are typically allowed?    A) Five players in the starting lineup and seven substitutes B) Five players in the starting lineup and five substitutes C) Seven players in the starting lineup and five substitutes D) Seven players in the starting lineup and seven substitutes   \\nSub-Question 1:   In the context of NBA and European Basketball League, how many players from each team are usually on the court at the start of the game?    A) Five players from each team B) Seven players from each team C) Four players from each team D) Six players from each team   \\nSub-Question 2:   Considering the NBA and FIBA rules, what is the usual number of allowed substitutes in a basketball game?    A) Five substitutes B) Seven substitutes C) Ten substitutes D) Twelve substitutes\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}