{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"68fe9b6a-e14c-5ad8-adcf-2551feeb9c45","task_key":"bbeh~5fdisambiguation~5fqa--full--68fe9b6a-e14c-5ad8-adcf-2551feeb9c45","task_revision_id":"1","upstream_id":"","short_description":"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived…","config":"bbeh_disambiguation_qa","split":"full","body":"{\"input\":\"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived from the context or is ambiguous.\\nSentence: The teacher met the engineer at the library where she was researching renewable energy and he was reading about marine animals. Afterwards, they walked to a nearby park where she explained that they were excited about the potential of wind power. He had some reservations.\\nWhich of the following options correctly explains the antecedent(s) of the pronoun(s)?\\n(A) The teacher was researching renewable energy and the engineer was reading about marine animals. \\n(B) The engineer was researching renewable energy and the teacher was reading about marine animals. \\n(C) The teacher was excited about wind power and the engineer had some reservations. \\n(D) The engineer was excited about wind power and the teacher had some reservations. \\n(E) Ambiguous.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}