{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9e99be72-d5cb-5103-b162-10cbd2cc9e4d","task_key":"bbeh~5fboardgame~5fqa--full--9e99be72-d5cb-5103-b162-10cbd2cc9e4d","task_revision_id":"1","upstream_id":"","short_description":"A few players are playing a boardgame. The current state of the game is as…","config":"bbeh_boardgame_qa","split":"full","body":"{\"input\":\"A few players are playing a boardgame. The current state of the game is as follows. The cobra reveals a secret to the songbird. The songbird brings an oil tank for the dachshund. The walrus does not surrender to the goose. And the rules of the game are as follows. Rule1: From observing that one animal surrenders to the chihuahua, one can conclude that it also hides her cards from the poodle, undoubtedly. Rule2: The badger unquestionably refuses to help the coyote, in the case where the swan tears down the castle of the badger. Rule3: If there is evidence that one animal, no matter which one, stops the victory of the coyote, then the cougar is not going to swim inside the pool located besides the house of the goose. Rule4: For the goose, if the belief is that the seal shouts at the goose and the cougar does not swim inside the pool located besides the house of the goose, then you can add \\\"the goose manages to convince the swan\\\" to your conclusions. Rule5: From observing that an animal neglects the bulldog, one can conclude the following: that animal does not hide the cards that she has from the poodle. Rule6: The goose unquestionably surrenders to the chihuahua, in the case where the walrus does not surrender to the goose. Rule7: The swan unquestionably tears down the castle of the badger, in the case where the goose does not manage to persuade the swan. Rule8: If something refuses to help the coyote, then it does not bring an oil tank for the reindeer. Rule9: If you are positive that you saw one of the animals brings an oil tank for the dachshund, you can be certain that it will also stop the victory of the coyote. Rule10: From observing that an animal hides her cards from the poodle, one can conclude the following: that animal does not manage to persuade the swan. Rule4 is preferred over Rule10. Rule5 is preferred over Rule1. A rule is only applicable if all of its antecedents can be proved. If a rule is preferred over the other, it means whenever both of them can be applied to derive new conclusions and those conclusions contradict with each other (e.g., from one we derive X and from the other we derive not X), we should go with the conclusion from the rule with higher preference. Based on the facts, rules, and preferences, what is the truth value of the statement,  does the badger bring an oil tank for the reindeer? Answer 'proved' if it can be proved, 'disproved' if it can be disproved, and 'unknown' if it can neither be proved nor disproved.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}