{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f9902054-00c8-5f8f-85c1-f34c941341df","task_key":"bbeh~5fdisambiguation~5fqa--full--f9902054-00c8-5f8f-85c1-f34c941341df","task_revision_id":"1","upstream_id":"","short_description":"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived…","config":"bbeh_disambiguation_qa","split":"full","body":"{\"input\":\"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived from the context or is ambiguous.\\nSentence: The scientist collaborated with the artist and he shared a story. She listened carefully and found the story implausible so she decided to ask some questions to verify if he is telling the truth. She asked his colleague whether he was present there at the time when all of that happened. Hearing this question, he got mad and decided to stop the collaboration.\\nWhich of the following options correctly explains the antecedent(s) of the pronoun(s)?\\n(A) The scientist shared a story. The colleague got mad.\\n(B) The scientist shared a story. The scientist got mad.\\n(C) The artist shared a story. The colleague got mad.\\n(D) The artist shared a story. The artist got mad.\\n(E) The scientist shared a story. It is ambiguous who got mad.\\n(F) The artist shared a story. It is ambiguous who got mad.\\n(G) It is ambiguous who shared a story. The scientist got mad.\\n(H) It is ambiguous who shared a story. The artist got mad.\\n(I) It is ambiguous who shared a story. The colleague got mad.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}