{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b31b939b-b0a0-547c-9af5-22c7dc5a195c","task_key":"bbeh~5fdisambiguation~5fqa--full--b31b939b-b0a0-547c-9af5-22c7dc5a195c","task_revision_id":"1","upstream_id":"","short_description":"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived…","config":"bbeh_disambiguation_qa","split":"full","body":"{\"input\":\"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived from the context or is ambiguous.\\nSentence: The tailor and the mechanic met at his house to discuss the project. He was pretty shocked to see a sewing machine there. Their meeting started later than expected because he got an important call right at the time the meeting was supposed to start.\\nWhich of the following options correctly explains the antecedent(s) of the pronoun(s)?\\n(A) The meeting was at the tailor's house. The mechanic was shocked. The mechanic got the call.\\n(B) The meeting was at the tailor’s house. It is ambiguous who was shocked. The mechanic got the call.\\n(C) The meeting was at the tailor’s house. It is ambiguous who was shocked. It is ambiguous who got the call.\\n(D) The meeting was at the mechanic’s house. The tailor was shocked. The tailor got the call.\\n(E) The meeting was at the mechanic’s house. The tailor was shocked. It is ambiguous who got the call.\\n(F) The meeting was at the mechanic’s house. It is ambiguous who was shocked. It is ambiguous who got the call.\\n(G) None of the above.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}