{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3eadc33e-cbaa-5732-abc0-19809fdae932","task_key":"bbeh~5fdisambiguation~5fqa--full--3eadc33e-cbaa-5732-abc0-19809fdae932","task_revision_id":"1","upstream_id":"","short_description":"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived…","config":"bbeh_disambiguation_qa","split":"full","body":"{\"input\":\"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived from the context or is ambiguous.\\nSentence: Sam asked Micah if she could help him with his homework. She said that she was busy but that he could ask Bill.\\nWhich of the following options correctly explains the antecedent(s) of the pronoun(s)?\\n(A) Sam needed help but Micah was busy. Micah said that Sam could ask Bill for help.\\n(B) Micah needed help but Sam was busy. Sam siad that Micah could ask Bill for help.\\n(C) Sam needed help but Micah was busy. Sam said Micah could ask Bill for help.\\n(D) Micah needed help but Sam was busy. Micah siad that Sam could ask Bill for help.\\n(E) None of the above.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}