{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"687fc469-c948-5edf-ae3c-9fb5cf30343b","task_key":"bbeh~5fdisambiguation~5fqa--full--687fc469-c948-5edf-ae3c-9fb5cf30343b","task_revision_id":"1","upstream_id":"","short_description":"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived…","config":"bbeh_disambiguation_qa","split":"full","body":"{\"input\":\"Here is a sentence with pronoun(s) whose antecedent(s) can either be derived from the context or is ambiguous.\\nSentence: Two siblings fight over their toys. He says she is ruining them all with her clumsiness. She complains that he never lets her play with anything, and it's not fair. The first parent, exasperated, walks in and asks what all the commotion is about. He insists that she's too rough. She will break the truck. The second parent, hearing the commotion from his study, emerges to see what's happening. She rolls her eyes and says he's just being mean.\\nWhich of the following options correctly explains the antecedent(s) of the pronoun(s)?\\n(A) The two siblings are a male and a female. The male sibling insists that the female sibling is rough. The first parent rolls her eyes. \\n(B) The two siblings are two males. The male sibling insists that the female sibling is rough. The female sibling rolls her eyes.\\n(C) The two siblings are a male and a female. The male sibling insists that the female is rough. It is ambiguous who, between the first parent and the female sibling, is rolling her eyes.\\n(D) The two siblings are a male and a female. The male sibling insists that the female sibling is rough. It is ambiguous who, between the first parent and the second parent, is rolling her eyes.\\n(E) The two siblings are a male and a female. The female sibling rolls her eyes. It is ambiguous who, between the male sibling and the first parent, is insisting.\\n(F) The two siblings are a male and a female. It is ambiguous who insists that the female sibling is rough and who rolls her eyes.\\n(G) The two siblings are a male and a female. The first parent is a female. It is ambiguous who insists that the female sibling is rough and who rolls her eyes.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}