{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"585f2ec8-35e3-5581-994a-7ba999a8044f","task_key":"bbeh~5fboardgame~5fqa--full--585f2ec8-35e3-5581-994a-7ba999a8044f","task_revision_id":"1","upstream_id":"","short_description":"A few players are playing a boardgame. The current state of the game is as…","config":"bbeh_boardgame_qa","split":"full","body":"{\"input\":\"A few players are playing a boardgame. The current state of the game is as follows. The bee creates one castle for the mannikin. The bulldog has 66 dollars. The cobra has 100 dollars, and has 15 friends. The cobra is watching a movie from 2023. The german shepherd has a flute, is watching a movie from 1978, and is a web developer. The mannikin has four friends. The otter acquires a photograph of the dinosaur. And the rules of the game are as follows. Rule1: There exists an animal which pays money to the vampire? Then the gadwall definitely enjoys the companionship of the duck. Rule2: There exists an animal which hugs the butterfly? Then the duck definitely takes over the emperor of the reindeer. Rule3: Regarding the cobra, if it is watching a movie that was released after Maradona died, then we can conclude that it refuses to help the akita. Rule4: From observing that one animal takes over the emperor of the reindeer, one can conclude that it also falls on a square of the chihuahua, undoubtedly. Rule5: If at least one animal swears to the walrus, then the akita does not negotiate a deal with the dalmatian. Rule6: If the akita negotiates a deal with the dalmatian, then the dalmatian stops the victory of the duck. Rule7: From observing that an animal does not surrender to the seahorse, one can conclude that it falls on a square of the beaver. Rule8: If the german shepherd has a musical instrument, then the german shepherd does not capture the king of the woodpecker. Rule9: Here is an important piece of information about the mannikin: if it has fewer than ten friends then it pays money to the vampire for sure. Rule10: The living creature that does not capture the king (i.e. the most important piece) of the woodpecker will reveal a secret to the butterfly with no doubts. Rule11: If at least one animal unites with the goat, then the duck does not fall on a square of the chihuahua. Rule12: One of the rules of the game is that if the gadwall does not enjoy the companionship of the duck, then the duck will never surrender to the seahorse. Rule13: For the akita, if the belief is that the otter swears to the akita and the cobra refuses to help the akita, then you can add \\\"the akita negotiates a deal with the dalmatian\\\" to your conclusions. Rule14: If something falls on a square of the chihuahua, then it borrows a weapon from the snake, too. Rule15: The duck unquestionably stops the victory of the cobra, in the case where the dalmatian stops the victory of the duck. Rule16: If you are positive that you saw one of the animals hides the cards that she has from the dinosaur, you can be certain that it will also swear to the akita. Rule17: One of the rules of the game is that if the dove does not suspect the truthfulness of the duck, then the duck will, without hesitation, surrender to the seahorse. Rule18: The rhino unquestionably destroys the wall constructed by the wolf, in the case where the gadwall suspects the truthfulness of the rhino. Rule19: If at least one animal borrows one of the weapons of the snake, then the gadwall suspects the truthfulness of the rhino. Rule12 is preferred over Rule17. Rule4 is preferred over Rule11. Rule5 is preferred over Rule13. A rule is only applicable if all of its antecedents can be proved. If a rule is preferred over the other, it means whenever both of them can be applied to derive new conclusions and those conclusions contradict with each other (e.g., from one we derive X and from the other we derive not X), we should go with the conclusion from the rule with higher preference. Based on the facts, rules, and preferences, what is the truth value of the statement,  does the rhino destroy the wall constructed by the wolf? Answer 'proved' if it can be proved, 'disproved' if it can be disproved, and 'unknown' if it can neither be proved nor disproved.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}