{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"12c88f08-e182-579c-9cbe-30cfc930b408","task_key":"bbeh~5fdisambiguation~5fqa--full--12c88f08-e182-579c-9cbe-30cfc930b408","task_revision_id":"1","upstream_id":"","short_description":"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived…","config":"bbeh_disambiguation_qa","split":"full","body":"{\"input\":\"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived from the context or is ambiguous.\\nSentence: The principal, the teacher and the student’s parents met to discuss his declining grades. She suggested a tutor and the parents scheduled weekly classes for the following weeks hoping it would improve his performance. They were very satisfied with the final result.\\nWhich of the following options correctly explains the antecedent(s) of the pronoun(s)?\\n(A) The principal suggested a tutor. The parents scheduled a meeting.\\n(B) The teacher suggested a tutor. The parents scheduled a meeting.\\n(C) The parents scheduled a meeting. The parents were satisfied with the result.\\n(D) The parents scheduled a meeting. The tutor and the teacher were satisfied with the results.\\n(E) It is ambiguous who suggested a tutor. It is ambiguous who was satisfied with the result. \\n(F) None of the above.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}