{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7716a150-3e5f-5cf3-a7fa-efde18594016","task_key":"bbeh~5fdisambiguation~5fqa--full--7716a150-3e5f-5cf3-a7fa-efde18594016","task_revision_id":"1","upstream_id":"","short_description":"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived…","config":"bbeh_disambiguation_qa","split":"full","body":"{\"input\":\"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived from the context or is ambiguous.\\nSentence: The chef and the food critic met at the newly opened restaurant where he asked the waiter to bring out the special tasting menu he had prepared. While he was explaining each dish's inspiration, she made notes meticulously, occasionally asking clarifying questions. After the dessert course, she praised the chef's creativity.\\nWhich of the following options correctly explains the antecedent(s) of the pronoun(s)?\\n(A) The chef prepared and brought out the dishes, the waiter explained each dish's inspiration, and the critic took notes.\\n(B) The waiter prepared the food and brought out the dishes, the chef explained each dish's inspiration, and the critic took notes.\\n(C) The chef prepared the food, the waiter brought out the dishes, the chef explained each dish's inspiration, and the critic took notes.\\n(D) The chef prepared the food, the waiter brought out the dishes, the critic explained each dish's inspiration, and the chef took notes.\\n(E) None of the above.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}