{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f552c9f5-0df6-57fb-a6aa-3d3cc5bf2c7d","task_key":"bbeh~5fdisambiguation~5fqa--full--f552c9f5-0df6-57fb-a6aa-3d3cc5bf2c7d","task_revision_id":"1","upstream_id":"","short_description":"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived…","config":"bbeh_disambiguation_qa","split":"full","body":"{\"input\":\"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived from the context or is ambiguous.\\nSentence: The teacher met the engineer at the library where she was researching renewable energy and he was reading about marine animals. Afterwards, they walked to a nearby park where he explained that they found penguins fascinating.\\nWhich of the following options correctly explains the antecedent(s) of the pronoun(s)?\\n(A) The teacher was researching renewable energy and the engineer was reading about marine animals. \\n(B) The engineer was researching renewable energy and the teacher was reading about marine animals. \\n(C) The teacher likes marine mammals. \\n(D) The engineer dislikes marine animals. \\n(E) Ambiguous.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}