{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6fac523e-0098-5a6e-8109-1534492f1a23","task_key":"bbeh~5fdisambiguation~5fqa--full--6fac523e-0098-5a6e-8109-1534492f1a23","task_revision_id":"1","upstream_id":"","short_description":"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived…","config":"bbeh_disambiguation_qa","split":"full","body":"{\"input\":\"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived from the context or is ambiguous.\\nSentence: The pathologist obtained tissue samples from the patient to look at under her microscope. She was then referred to the surgeon because she had a rare skin condition. After the examination, they warned her that she needed to get more rest.\\nWhich of the following options correctly explains the antecedent(s) of the pronoun(s)?\\n(A) It was the pathologist’s microscope and the surgeon warned that the patient needed more rest.\\n(B) It was the patient’s microscope and the pathologist warned that the patient needed more rest.\\n(C) The pathologist had a skin condition and the surgeon warned that the pathologist needed more rest.\\n(D) It was the pathologist’s microscope and the pathologist warned that the patient needed more rest.\\n(E) None of the above.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}