{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"123353ab-f1dc-5e02-b163-f9d134fb5fec","task_key":"bbeh~5fdisambiguation~5fqa--full--123353ab-f1dc-5e02-b163-f9d134fb5fec","task_revision_id":"1","upstream_id":"","short_description":"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived…","config":"bbeh_disambiguation_qa","split":"full","body":"{\"input\":\"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived from the context or is ambiguous.\\nSentence: The stay-at-home dad was preparing dinner for his husband who was working late. He chopped vegetables, stirred sauces, and set the table. He knew he would be tired after a long day at the office, and he wanted to have a delicious meal waiting. He looked forward to his return and a relaxing evening together.\\nWhich of the following options correctly explains the antecedent(s) of the pronoun(s)?\\n(A) The stay-at-home prepares the dinner. The husband would be tired. The husband wanted to have a meal waiting. The husband looks forward to going back home.\\n(B) The husband prepares the dinner. The stay-at-home dad would be tired. The husband wanted to have a meal waiting. The husband looks forward to going back home.\\n(C) The husband prepares the dinner. The stay-at-home dad would be tired. It is ambiguous who wants to have a meal waiting. The stay-at-home dad looks forward to the husband being back home.\\n(D) The stay-at-home prepares the dinner. The husband would be tired. It is ambiguous if the husband looks forward to going back home or the stay-at-home dad looks forward to the husband being back home.\\n(E) It is ambiguous who prepares the dinner. The husband would be tired. It is ambiguous if the husband looks forward to going back home or the stay-at-home dad looks forward to the husband being back home.\\n(F) None of the above.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}