{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5bb23bff-eb2f-58ec-bbde-14fcf81ec714","task_key":"bbeh~5fsportqa--full--5bb23bff-eb2f-58ec-bbde-14fcf81ec714","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: How does the combination of helmet certification standards and visor tint regulations affect player equipment compliance in American football during officially sanctioned games?  A) The helmet must meet NOCSAE standards and have a visible certification sticker. B) Visors must be clear to allow for medical personnel to see a player's eyes without removal. C) Visors can have a light tint as long as they are not reflective. D) Helmets must be reconditioned every year to maintain the NOCSAE certification.\\nSub-Question 1: How do helmet certification requirements influence player safety and equipment checks prior to an American football game?  A) Players must present their helmets for a safety check by officials before the game. B) Helmets without a visible NOCSAE certification sticker are not permitted. C) Helmets can be any model as long as they are less than 10 years old. D) Players are responsible for ensuring their helmets are NOCSAE certified.\\nSub-Question 2: How do visor regulations impact player vision and safety during gameplay in American football?  A) Visors must be made of shatterproof material to be approved for use. B) Any visor tint is allowed as long as the player has a medical exemption. C) Reflective visors are permitted during night games for better visibility. D) Clear visors are mandatory unless a documented medical condition necessitates a tinted visor.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}