{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c134444c-e5ba-5925-9cb0-a955bd6a9d9e","task_key":"bbeh~5fdisambiguation~5fqa--full--c134444c-e5ba-5925-9cb0-a955bd6a9d9e","task_revision_id":"1","upstream_id":"","short_description":"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived…","config":"bbeh_disambiguation_qa","split":"full","body":"{\"input\":\"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived from the context or is ambiguous.\\nSentence: During the meeting the manager presented to the team the plan for the next quarter. One member of the team seemed anxious, and the manager tried to assuage his worries. Another member of the team said that they also think his plan is too ambitious and might not work.\\nWhich of the following options correctly explains the antecedent(s) of the pronoun(s)?\\n(A) The meeting has at most one male participant.\\n(B) It is ambiguous whether the manager is male or female, and one male member of the team thinks the manager’s plan is too ambitious.\\n(C) The meeting had at least three male participants (possibly including the manager).\\n(D) The manager is male, and tried to assuage the worries of a male participant in the meeting.\\n(E) There are definitely only two male participants in the meeting.\\n(F) None of the above.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}