{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5409d5d4-7725-55e3-8754-a4759f37e413","task_key":"bbeh~5fdisambiguation~5fqa--full--5409d5d4-7725-55e3-8754-a4759f37e413","task_revision_id":"1","upstream_id":"","short_description":"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived…","config":"bbeh_disambiguation_qa","split":"full","body":"{\"input\":\"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived from the context or is ambiguous.\\nSentence: The salesperson bumped into a famous writer. He thought him a fool so tried to sell him trinkets that were worthless. However his ploy was seen through when the old typewriter he claimed to have written his past famous books on did not work at all.\\nWhich of the following options correctly explains the antecedent(s) of the pronoun(s)?\\n(A) The salesperson was trying to sell the writer a typewriter that the salesperson claim he wrote books with.\\n(B) The famous writer was trying to sell the salesperson a typewriter that the writer claim he wrote his famous books with.\\n(C) Ambiguous.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}