{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"093276b9-6194-5083-97d3-fb5e34b24d66","task_key":"bbeh~5fboardgame~5fqa--full--093276b9-6194-5083-97d3-fb5e34b24d66","task_revision_id":"1","upstream_id":"","short_description":"A few players are playing a boardgame. The current state of the game is as…","config":"bbeh_boardgame_qa","split":"full","body":"{\"input\":\"A few players are playing a boardgame. The current state of the game is as follows. The coyote reduced her work hours recently. The fish has a basketball with a diameter of 27 inches, and has a couch. The liger manages to convince the coyote. The seal surrenders to the swan. The duck does not trade one of its pieces with the pelikan. And the rules of the game are as follows. Rule1: The coyote unquestionably swims in the pool next to the house of the chihuahua, in the case where the liger manages to convince the coyote. Rule2: This is a basic rule: if the seahorse takes over the emperor of the coyote, then the conclusion that \\\"the coyote will not swim in the pool next to the house of the chihuahua\\\" follows immediately and effectively. Rule3: If the butterfly disarms the otter, then the otter reveals a secret to the mermaid. Rule4: The living creature that suspects the truthfulness of the llama will also destroy the wall constructed by the cobra, without a doubt. Rule5: The living creature that swims in the pool next to the house of the chihuahua will also refuse to help the llama, without a doubt. Rule6: There exists an animal which borrows one of the weapons of the gorilla? Then the coyote definitely tears down the castle of the dragon. Rule7: There exists an animal which reveals something that is supposed to be a secret to the mermaid? Then, the dinosaur definitely does not manage to persuade the dachshund. Rule8: If the duck swears to the badger, then the badger hides her cards from the rhino. Rule9: If there is evidence that one animal, no matter which one, destroys the wall built by the cobra, then the stork suspects the truthfulness of the gadwall undoubtedly. Rule10: The fish will call the chinchilla if it (the fish) has a basketball that fits in a 30.1 x 25.3 x 28.5 inches box. Rule11: One of the rules of the game is that if the seahorse shouts at the coyote, then the coyote will never destroy the wall constructed by the cobra. Rule12: From observing that one animal calls the chinchilla, one can conclude that it also builds a power plant close to the green fields of the ostrich, undoubtedly. Rule13: The living creature that does not tear down the castle that belongs to the dragon will manage to persuade the mannikin with no doubts. Rule14: Be careful when something refuses to help the llama and also manages to convince the mannikin because in this case it will surely suspect the truthfulness of the llama (this may or may not be problematic). Rule15: There exists an animal which hides the cards that she has from the rhino? Then the ostrich definitely unites with the goat. Rule16: If something does not smile at the dugong, then it manages to persuade the dachshund. Rule17: If there is evidence that one animal, no matter which one, negotiates a deal with the peafowl, then the coyote is not going to refuse to help the llama. Rule18: In order to conclude that the dachshund will never swear to the zebra, two pieces of evidence are required: firstly the german shepherd should invest in the company whose owner is the dachshund and secondly the dinosaur should not manage to convince the dachshund. Rule19: If there is evidence that one animal, no matter which one, suspects the truthfulness of the gadwall, then the german shepherd invests in the company whose owner is the dachshund undoubtedly. Rule20: If there is evidence that one animal, no matter which one, unites with the goat, then the butterfly disarms the otter undoubtedly. Rule21: Regarding the coyote, if it works fewer hours than before, then we can conclude that it does not tear down the castle of the dragon. Rule22: One of the rules of the game is that if the owl refuses to help the badger, then the badger will never hide her cards from the rhino. Rule23: The living creature that does not trade one of its pieces with the pelikan will swear to the badger with no doubts. Rule24: If the fish has something to sit on, then the fish calls the chinchilla. Rule11 is preferred over Rule4. Rule16 is preferred over Rule7. Rule17 is preferred over Rule5. Rule2 is preferred over Rule1. Rule22 is preferred over Rule8. Rule6 is preferred over Rule21. A rule is only applicable if all of its antecedents can be proved. If a rule is preferred over the other, it means whenever both of them can be applied to derive new conclusions and those conclusions contradict with each other (e.g., from one we derive X and from the other we derive not X), we should go with the conclusion from the rule with higher preference. Based on the facts, rules, and preferences, what is the truth value of the statement,  does the dachshund swear to the zebra? Answer 'proved' if it can be proved, 'disproved' if it can be disproved, and 'unknown' if it can neither be proved nor disproved.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}