{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"115ef3bd-d397-568e-a4e6-d4e3b2a68691","task_key":"bbeh~5fboardgame~5fqa--full--115ef3bd-d397-568e-a4e6-d4e3b2a68691","task_revision_id":"1","upstream_id":"","short_description":"A few players are playing a boardgame. The current state of the game is as…","config":"bbeh_boardgame_qa","split":"full","body":"{\"input\":\"A few players are playing a boardgame. The current state of the game is as follows. The camel destroys the wall constructed by the seahorse. The flamingo assassinated the mayor. The flamingo has a 14 x 10 inches notebook. The frog surrenders to the flamingo. The gadwall pays money to the goat. The owl surrenders to the swan. The songbird negotiates a deal with the dugong. The vampire acquires a photograph of the walrus. And the rules of the game are as follows. Rule1: If at least one animal acquires a photograph of the walrus, then the goose calls the flamingo. Rule2: If something swears to the leopard and does not unite with the vampire, then it suspects the truthfulness of the pigeon. Rule3: The swan does not refuse to help the flamingo, in the case where the owl surrenders to the swan. Rule4: If something hugs the butterfly, then it neglects the finch, too. Rule5: One of the rules of the game is that if the seal disarms the peafowl, then the peafowl will, without hesitation, manage to convince the otter. Rule6: From observing that an animal destroys the wall constructed by the duck, one can conclude the following: that animal does not trade one of its pieces with the fish. Rule7: The living creature that smiles at the dinosaur will never enjoy the company of the beaver. Rule8: If the goose calls the flamingo and the swan does not refuse to help the flamingo, then the flamingo will never unite with the vampire. Rule9: If at least one animal refuses to help the chihuahua, then the seal disarms the peafowl. Rule10: If at least one animal suspects the truthfulness of the pigeon, then the swallow neglects the stork. Rule11: One of the rules of the game is that if the frog surrenders to the flamingo, then the flamingo will, without hesitation, acquire a photograph of the snake. Rule12: If you are positive that you saw one of the animals neglects the stork, you can be certain that it will also smile at the dinosaur. Rule13: One of the rules of the game is that if the dugong calls the worm, then the worm will, without hesitation, bring an oil tank for the llama. Rule14: If the german shepherd unites with the dove, then the dove is not going to hide her cards from the swallow. Rule15: If the peafowl trades one of the pieces in its possession with the fish, then the fish refuses to help the chihuahua. Rule16: The camel unquestionably hugs the butterfly, in the case where the seahorse creates a castle for the camel. Rule17: If something acquires a photograph of the snake, then it swears to the leopard, too. Rule18: If there is evidence that one animal, no matter which one, brings an oil tank for the llama, then the mermaid acquires a photograph of the vampire undoubtedly. Rule19: The peafowl trades one of the pieces in its possession with the fish whenever at least one animal pays some $$$ to the goat. Rule20: The seahorse unquestionably creates one castle for the camel, in the case where the camel destroys the wall constructed by the seahorse. Rule21: If there is evidence that one animal, no matter which one, negotiates a deal with the dugong, then the bulldog takes over the emperor of the dugong undoubtedly. Rule22: If there is evidence that one animal, no matter which one, borrows a weapon from the bison, then the peafowl is not going to manage to convince the otter. Rule23: If the bulldog takes over the emperor of the dugong, then the dugong calls the worm. Rule24: Here is an important piece of information about the flamingo: if it voted for the mayor then it swims inside the pool located besides the house of the ant for sure. Rule25: Here is an important piece of information about the flamingo: if it has a notebook that fits in a 15.5 x 17.2 inches box then it swims in the pool next to the house of the ant for sure. Rule26: There exists an animal which neglects the finch? Then the german shepherd definitely unites with the dove. Rule22 is preferred over Rule5. Rule6 is preferred over Rule19. A rule is only applicable if all of its antecedents can be proved. If a rule is preferred over the other, it means whenever both of them can be applied to derive new conclusions and those conclusions contradict with each other (e.g., from one we derive X and from the other we derive not X), we should go with the conclusion from the rule with higher preference. Based on the facts, rules, and preferences, what is the truth value of the statement,  does the swallow enjoy the company of the beaver? Answer 'proved' if it can be proved, 'disproved' if it can be disproved, and 'unknown' if it can neither be proved nor disproved.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}