{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"87291b4b-0eaa-5a17-bf20-890e05ce1072","task_key":"bbeh~5fdisambiguation~5fqa--full--87291b4b-0eaa-5a17-bf20-890e05ce1072","task_revision_id":"1","upstream_id":"","short_description":"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived…","config":"bbeh_disambiguation_qa","split":"full","body":"{\"input\":\"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived from the context or is ambiguous.\\nSentence: The lawyer representing the defendant in the case was known for his aggressive tactics. He often argued with the judge and opposing counsel, which frustrated the client. Despite this, he was confident that he could win the case. The jury, however, was ultimately unconvinced.\\nWhich of the following options correctly explains the antecedent(s) of the pronoun(s)?\\n(A) The judge was frustrated with the lawyer. \\n(B) The client was confident that he could win.\\n(C) The lawyer was confident that he could win.\\n(D) It is ambiguous who was confident he could win.\\n(E) The client had aggressive tactics.\\n(F) Ambiguous.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}