{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"27be03ec-f16e-5faf-8fc4-d825df5ef162","task_key":"bbeh~5fsportqa--full--27be03ec-f16e-5faf-8fc4-d825df5ef162","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question:  During a kick-off in a football match, if a team A's striker is at the center circle of the pitch, the team's goalkeeper is at the goal post, and the team's defenders are in their own half of the field, where should team A's midfielders be positioned considering the rules of football during a kick-off?   A) In the opponent's half of the field B) In their own half of the field C) At the center circle of the pitch D) At the goal post\\nSub-Question 1:  According to football rules, where should all players be positioned during a kick-off?  A) In their own half of the field B) In the opponent's half of the field C) Anywhere on the pitch D) In the center circle of the pitch\\nSub-Question 2:  Where is the center circle of the pitch located in relation to the midfielders' position during a kick-off?  A) It is in their own half of the field B) It is in the opponent's half of the field C) It is at the goal post D) It is the boundary between their own half and the opponent's half of the field\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}