{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"96dca2bf-9c5e-5a15-b558-6b5859fb0a20","task_key":"bbeh~5fboardgame~5fqa--full--96dca2bf-9c5e-5a15-b558-6b5859fb0a20","task_revision_id":"1","upstream_id":"","short_description":"A few players are playing a boardgame. The current state of the game is as…","config":"bbeh_boardgame_qa","split":"full","body":"{\"input\":\"A few players are playing a boardgame. The current state of the game is as follows. The bear struggles to find food. The bulldog manages to convince the lizard. The chinchilla enjoys the company of the basenji. The crab is named Max. The crab manages to convince the seahorse. The mannikin enjoys the company of the dugong. The mermaid has 61 dollars. The otter is named Peddi. The owl has 28 dollars, and supports Chris Ronaldo. The owl is named Pablo. The pelikan is named Mojo. And the rules of the game are as follows. Rule1: If the mannikin enjoys the company of the dugong, then the dugong hides her cards from the seahorse. Rule2: This is a basic rule: if the bulldog manages to convince the lizard, then the conclusion that \\\"the lizard hides her cards from the goose\\\" follows immediately and effectively. Rule3: There exists an animal which shouts at the liger? Then, the rhino definitely does not swim inside the pool located besides the house of the ostrich. Rule4: If something neglects the wolf and trades one of its pieces with the crab, then it creates one castle for the woodpecker. Rule5: This is a basic rule: if the cobra tears down the castle that belongs to the peafowl, then the conclusion that \\\"the peafowl will not create a castle for the woodpecker\\\" follows immediately and effectively. Rule6: The living creature that creates a castle for the goat will never neglect the wolf. Rule7: If there is evidence that one animal, no matter which one, borrows one of the weapons of the dinosaur, then the woodpecker pays money to the husky undoubtedly. Rule8: If at least one animal disarms the chihuahua, then the dugong tears down the castle of the lizard. Rule9: The seahorse unquestionably trades one of the pieces in its possession with the swan, in the case where the dugong hides the cards that she has from the seahorse. Rule10: If you are positive that one of the animals does not hide her cards from the goose, you can be certain that it will borrow a weapon from the dinosaur without a doubt. Rule11: If the bear has difficulty to find food, then the bear calls the seahorse. Rule12: The peafowl creates a castle for the woodpecker whenever at least one animal refuses to help the ostrich. Rule13: If at least one animal tears down the castle of the lizard, then the swallow calls the dove. Rule14: This is a basic rule: if the bear calls the seahorse, then the conclusion that \\\"the seahorse trades one of the pieces in its possession with the crab\\\" follows immediately and effectively. Rule15: The owl will not enjoy the company of the seahorse if it (the owl) is a fan of Chris Ronaldo. Rule16: The rhino unquestionably swims in the pool next to the house of the ostrich, in the case where the chinchilla dances with the rhino. Rule17: One of the rules of the game is that if the crab brings an oil tank for the seahorse, then the seahorse will, without hesitation, neglect the wolf. Rule18: For the woodpecker, if the belief is that the peafowl creates one castle for the woodpecker and the seahorse creates a castle for the woodpecker, then you can add \\\"the woodpecker disarms the chihuahua\\\" to your conclusions. Rule19: If something enjoys the companionship of the basenji, then it dances with the rhino, too. Rule20: Here is an important piece of information about the crab: if it has a name whose first letter is the same as the first letter of the pelikan's name then it brings an oil tank for the seahorse for sure. Rule21: Here is an important piece of information about the owl: if it has more money than the mermaid then it does not enjoy the company of the seahorse for sure. Rule3 is preferred over Rule16. Rule5 is preferred over Rule12. Rule6 is preferred over Rule17. A rule is only applicable if all of its antecedents can be proved. If a rule is preferred over the other, it means whenever both of them can be applied to derive new conclusions and those conclusions contradict with each other (e.g., from one we derive X and from the other we derive not X), we should go with the conclusion from the rule with higher preference. Based on the facts, rules, and preferences, what is the truth value of the statement,  does the swallow call the dove? Answer 'proved' if it can be proved, 'disproved' if it can be disproved, and 'unknown' if it can neither be proved nor disproved.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}