{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5e588f76-a4ee-57be-a08c-d38eab8323ec","task_key":"bbeh~5fdisambiguation~5fqa--full--5e588f76-a4ee-57be-a08c-d38eab8323ec","task_revision_id":"1","upstream_id":"","short_description":"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived…","config":"bbeh_disambiguation_qa","split":"full","body":"{\"input\":\"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived from the context or is ambiguous.\\nSentence: Janet and John went on a road trip. She had planned the itinerary meticulously, marking points of interest and scenic routes. He was in charge of navigation and entertainment, ensuring they never got lost and that there was always good music playing. They both looked forward to the adventures that awaited them.\\nWhich of the following options correctly explains the antecedent(s) of the pronoun(s)?\\n(A) John planned the itinerary. Janet was in charge of navigation.\\n(B) Janet planned the itinerary. John was in charge of navigation.\\n(C) John planned the itinerary and was in charge of navigation. Mary looked forward to the trip.\\n(D) Mary planned the itinerary and was in charge of navigation. John looked forward to the trip.\\n(E) None of the above.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}