{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4f2da9da-ec96-556b-ac51-b96f592a9202","task_key":"bbeh~5fsportqa--full--4f2da9da-ec96-556b-ac51-b96f592a9202","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question:   In a high-stakes volleyball match, what decisions might a setter make while executing offensive strategies to increase the chances of scoring?    A) The setter might decide to set the ball to the power hitter who is in the best position to score. B) The setter might decide to set a quick ball to the middle hitter, catching the opponent's defense off-guard. C) The setter might decide to set a back row attack to diversify the offensive options. D) The setter might decide to serve the ball directly to the opponent's weakest receiver.   \\nSub-Question 1:   What factors could influence the setter's decision to set the ball to the power hitter who is in the best position to score?    A) The current defensive formation of the opposing team. B) The physical condition and performance of the power hitter in the current game. C) The power hitter's historical performance against the current opposing team. D) The setter's personal preference or bias towards the power hitter.   \\nSub-Question 2:   What factors could influence the setter's decision to set a quick ball to the middle hitter, catching the opponent's defense off-guard?    A) The middle hitter's readiness and position relative to the net. B) The current positioning and reaction speed of the opponent's blockers. C) The setter's confidence in the middle hitter's ability to execute a quick attack. D) The success rate of quick attacks in the current game.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}