{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"08a345ef-e7e7-592f-bc47-6e6d081fe332","task_key":"bbeh~5fsportqa--full--08a345ef-e7e7-592f-bc47-6e6d081fe332","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: Which type of visor is allowed in a professional American football helmet during official NFL games, considering the regulations regarding the material's transparency and the league's approved list of equipment providers?  A) A tinted visor from a non-approved equipment provider. B) A clear visor produced by an NFL-approved equipment provider. C) A reflective visor that has been certified by the player's optometrist. D) A polarized visor designed for high-contrast visibility.\\nSub-Question 1: Which characteristic is mandated by the NFL for helmet visors used during official games?  A) They must be polarized. B) They must be tinted. C) They must be clear. D) They must be reflective.\\nSub-Question 2: Who is eligible to approve American football equipment providers for the NFL?  A) The team's head coach. B) The NFL Equipment Manager's office. C) The player's personal trainer. D) The American Football Coaches Association.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}