{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"401f4a29-49a1-5624-b2e3-cc98f0d1ee85","task_key":"bbeh~5fdisambiguation~5fqa--full--401f4a29-49a1-5624-b2e3-cc98f0d1ee85","task_revision_id":"1","upstream_id":"","short_description":"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived…","config":"bbeh_disambiguation_qa","split":"full","body":"{\"input\":\"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived from the context or is ambiguous.\\nSentence: They previously agreed on the phone that the customer can pay with cash. But when the electrician showed up at his home, he only had large bills and she didn’t have enough loose change, so in the end had to go to the ATM to get the exact cash to pay her.\\nWhich of the following options correctly explains the antecedent(s) of the pronoun(s)?\\n(A) The electrician went to the ATM because the customer didn’t have enough loose change.\\n(B) The customer went to the ATM because the electrician didn’t have enough loose change.\\n(C) The electrician went to the ATM because the electrician didn’t have enough loose change.\\n(D) The customer went to the ATM because the customer didn’t have enough loose change.\\n(E) Ambiguous.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}