{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f127f806-d500-53be-bd22-8961a457b5e5","task_key":"bbeh~5fsportqa--full--f127f806-d500-53be-bd22-8961a457b5e5","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question:  Why was a yellow card issued to Player A in the football match between Team X and Team Y?  A) Player A committed a reckless challenge on Player B. B) Player A persistently infringed the Laws of the Game. C) Player A deliberately left the field of play without the referee’s permission. D) Player A showed dissent by word or action.\\nSub-Question 1:  Why is a reckless challenge considered as a yellow card offense?  A) It endangers the safety of an opponent. B) It violates the spirit of the game. C) It disrupts the flow of the game. D) It demonstrates a lack of respect for the game.\\nSub-Question 2:  Why is persistently infringing the Laws of the Game considered as a yellow card offense?  A) It shows a lack of discipline and respect for the rules. B) It disrupts the flow of the game. C) It gives the offending team an unfair advantage. D) It negatively impacts the enjoyment of spectators.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}