{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"00878d02-f8c5-5341-a8f1-fe3525f296b4","task_key":"bbeh~5fdisambiguation~5fqa--full--00878d02-f8c5-5341-a8f1-fe3525f296b4","task_revision_id":"1","upstream_id":"","short_description":"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived…","config":"bbeh_disambiguation_qa","split":"full","body":"{\"input\":\"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived from the context or is ambiguous.\\nSentence: The teacher and the school principal met in her office where he presented a proposal for a new after-school club focused on robotics. She listened attentively, asking questions about the resources required and the potential student interest. She expressed initial support but requested a more detailed budget breakdown. He agreed to provide it.\\nWhich of the following options correctly explains the antecedent(s) of the pronoun(s)?\\n(A) It was the teacher's office and the principal presented the proposal.\\n(B) It was the principal's office and the teacher presented the proposal.\\n(C) It is ambiguous whose office it was and the teacher presented the proposal.\\n(D) It is ambiguous whose office it was and the principal presented the proposal.\\n(E) It was the teacher's office and it is unclear who presented the proposal.\\n(F) It was the principal's office and it is unclear who presented the proposal.\\n(G) None of the above.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}