{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7232d036-b863-5791-a9fe-2c303fcd2b04","task_key":"bbeh~5fboardgame~5fqa--full--7232d036-b863-5791-a9fe-2c303fcd2b04","task_revision_id":"1","upstream_id":"","short_description":"A few players are playing a boardgame. The current state of the game is as…","config":"bbeh_boardgame_qa","split":"full","body":"{\"input\":\"A few players are playing a boardgame. The current state of the game is as follows. The cobra neglects the coyote. The poodle builds a power plant near the green fields of the bear. The rhino swears to the poodle. The songbird reveals a secret to the swallow. The dragon does not swim in the pool next to the house of the poodle. And the rules of the game are as follows. Rule1: The living creature that calls the flamingo will never suspect the truthfulness of the owl. Rule2: If something disarms the chihuahua, then it swears to the gorilla, too. Rule3: If the cobra neglects the coyote, then the coyote tears down the castle of the dinosaur. Rule4: If something tears down the castle that belongs to the dinosaur, then it swears to the mouse, too. Rule5: From observing that one animal builds a power plant close to the green fields of the bear, one can conclude that it also disarms the ostrich, undoubtedly. Rule6: From observing that an animal trades one of its pieces with the dalmatian, one can conclude the following: that animal does not swear to the mouse. Rule7: If the lizard captures the king of the beaver, then the beaver disarms the chihuahua. Rule8: The beaver does not swear to the gorilla whenever at least one animal refuses to help the gadwall. Rule9: Be careful when something negotiates a deal with the german shepherd and also falls on a square that belongs to the mule because in this case it will surely not refuse to help the lizard (this may or may not be problematic). Rule10: If the mouse does not refuse to help the lizard, then the lizard captures the king (i.e. the most important piece) of the beaver. Rule11: This is a basic rule: if the cougar builds a power plant close to the green fields of the mouse, then the conclusion that \\\"the mouse will not negotiate a deal with the german shepherd\\\" follows immediately and effectively. Rule12: If at least one animal reveals a secret to the swallow, then the mouse suspects the truthfulness of the owl. Rule13: There exists an animal which creates one castle for the husky? Then, the mouse definitely does not fall on a square of the mule. Rule14: From observing that one animal suspects the truthfulness of the owl, one can conclude that it also falls on a square that belongs to the mule, undoubtedly. Rule15: If there is evidence that one animal, no matter which one, disarms the ostrich, then the mouse negotiates a deal with the german shepherd undoubtedly. Rule1 is preferred over Rule12. Rule11 is preferred over Rule15. Rule13 is preferred over Rule14. Rule6 is preferred over Rule4. Rule8 is preferred over Rule2. A rule is only applicable if all of its antecedents can be proved. If a rule is preferred over the other, it means whenever both of them can be applied to derive new conclusions and those conclusions contradict with each other (e.g., from one we derive X and from the other we derive not X), we should go with the conclusion from the rule with higher preference. Based on the facts, rules, and preferences, what is the truth value of the statement,  does the beaver swear to the gorilla? Answer 'proved' if it can be proved, 'disproved' if it can be disproved, and 'unknown' if it can neither be proved nor disproved.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}