{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1528d15c-4905-5908-b306-79ec0c06beeb","task_key":"bbeh~5fsportqa--full--1528d15c-4905-5908-b306-79ec0c06beeb","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question:  Which of the following scenarios in a volleyball game are considered losing a point?    A) A player hits the ball twice in succession. B) A player performs a service outside the service zone. C) The ball hits the antenna during a rally. D) A player touches the net while attempting to block a shot.   \\nSub-Question 1:  In which of the following scenarios would a double hit by a player be acceptable?    A) When performing a spike. B) When making a first team contact. C) When performing a block. D) When serving the ball.   \\nSub-Question 2:  Which of these actions are considered violations when performing a service in volleyball?    A) The server steps on the end line before the ball is hit. B) The ball is served underhand. C) The server tosses the ball into the air and lets it drop without hitting it. D) The server hits the ball with two hands.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}