{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"54604cb0-3e1a-5173-a595-59bc455079ce","task_key":"bbeh~5fboardgame~5fqa--full--54604cb0-3e1a-5173-a595-59bc455079ce","task_revision_id":"1","upstream_id":"","short_description":"A few players are playing a boardgame. The current state of the game is as…","config":"bbeh_boardgame_qa","split":"full","body":"{\"input\":\"A few players are playing a boardgame. The current state of the game is as follows. The rhino is currently in Ottawa. The stork stops the victory of the rhino. And the rules of the game are as follows. Rule1: If the butterfly disarms the rhino and the stork stops the victory of the rhino, then the rhino will not enjoy the company of the badger. Rule2: From observing that an animal wants to see the pigeon, one can conclude the following: that animal does not borrow a weapon from the bulldog. Rule3: From observing that an animal does not enjoy the company of the badger, one can conclude the following: that animal will not borrow one of the weapons of the dragon. Rule4: Regarding the rhino, if it is in Canada at the moment, then we can conclude that it enjoys the company of the badger. Rule5: If there is evidence that one animal, no matter which one, enjoys the company of the badger, then the shark borrows one of the weapons of the dragon undoubtedly. Rule6: This is a basic rule: if the pigeon wants to see the dugong, then the conclusion that \\\"the dugong wants to see the pigeon\\\" follows immediately and effectively. Rule7: The living creature that swears to the gadwall will never manage to persuade the peafowl. Rule8: There exists an animal which manages to convince the peafowl? Then the pigeon definitely wants to see the dugong. Rule9: The songbird manages to persuade the peafowl whenever at least one animal borrows a weapon from the dragon. Rule10: From observing that an animal does not neglect the gorilla, one can conclude the following: that animal will not want to see the dugong. Rule1 is preferred over Rule4. Rule10 is preferred over Rule8. Rule3 is preferred over Rule5. Rule7 is preferred over Rule9. A rule is only applicable if all of its antecedents can be proved. If a rule is preferred over the other, it means whenever both of them can be applied to derive new conclusions and those conclusions contradict with each other (e.g., from one we derive X and from the other we derive not X), we should go with the conclusion from the rule with higher preference. Based on the facts, rules, and preferences, what is the truth value of the statement,  does the dugong borrow one of the weapons of the bulldog? Answer 'proved' if it can be proved, 'disproved' if it can be disproved, and 'unknown' if it can neither be proved nor disproved.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}