{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cb9090bb-7010-5de8-b198-44fe7a87debc","task_key":"bbeh~5fsportqa--full--cb9090bb-7010-5de8-b198-44fe7a87debc","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: Which player would most likely be able to execute a critical play during the final minutes of a tightly contested playoff game, considering their performance in high-pressure situations throughout the current season and their historical playoff experience?  A) The quarterback with the highest number of fourth-quarter comebacks this season but no prior playoff experience. B) The wide receiver with the most touchdown receptions in the regular season who has also won a Super Bowl MVP in the past. C) The running back with the highest yards per carry average in the last four games but with only one previous playoff appearance. D) The tight end who has consistently been the second option on third downs and has multiple playoff game-winning touchdowns.\\nSub-Question 1: Which player has demonstrated the ability to perform under high pressure by leading fourth-quarter comebacks in the current season?  A) The quarterback B) The wide receiver C) The running back D) The tight end\\nSub-Question 2: Considering historical playoff experiences and the ability to make critical plays, which player has been recognized as the most valuable in a Super Bowl?  A) The quarterback B) The wide receiver C) The running back D) The tight end\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}