{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ffc5d3aa-6672-5c4d-a23d-6fd7f6ed533e","task_key":"bbeh~5fsportqa--full--ffc5d3aa-6672-5c4d-a23d-6fd7f6ed533e","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question:   In a basketball match, Player A receives the ball, takes two steps without dribbling, then dribbles, stops, and dribbles again before passing it to Player B. Player B, after receiving the ball, takes three steps without dribbling before making a shot. Why might these actions be considered violations according to standard basketball rules?  A) Player A committed a double dribble and Player B committed traveling. B) Player A committed traveling and Player B committed a double dribble. C) Both Player A and Player B committed traveling. D) Both Player A and Player B committed a double dribble.    \\nSub-Question 1:   Why might Player A's action be considered a violation? A) Player A took two steps without dribbling. B) Player A dribbled, stopped, and dribbled again. C) Player A passed the ball to Player B. D) Player A dribbled after taking two steps.   \\nSub-Question 2:   Why might Player B's action be considered a violation?  A) Player B took three steps without dribbling. B) Player B received the ball from Player A. C) Player B made a shot after taking three steps. D) Player B dribbled before making a shot.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}