{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"17bbdf3a-e11c-583a-9356-09e21858eebb","task_key":"bbeh~5fboardgame~5fqa--full--17bbdf3a-e11c-583a-9356-09e21858eebb","task_revision_id":"1","upstream_id":"","short_description":"A few players are playing a boardgame. The current state of the game is as…","config":"bbeh_boardgame_qa","split":"full","body":"{\"input\":\"A few players are playing a boardgame. The current state of the game is as follows. The bear is currently in Brazil, and is twenty months old. The dinosaur has 34 dollars. The dragonfly has 51 dollars. The finch struggles to find food. The goat has 68 dollars, and is currently in Brazil. The goat struggles to find food. The mannikin has 32 dollars. And the rules of the game are as follows. Rule1: This is a basic rule: if the fangtooth neglects the mermaid, then the conclusion that \\\"the mermaid will not refuse to help the liger\\\" follows immediately and effectively. Rule2: Regarding the bear, if it is less than 4 years old, then we can conclude that it shouts at the bulldog. Rule3: Here is an important piece of information about the dinosaur: if it has more money than the mannikin then it unites with the seal for sure. Rule4: The living creature that borrows one of the weapons of the seal will never take over the emperor of the dragonfly. Rule5: From observing that an animal does not swear to the woodpecker, one can conclude that it enjoys the companionship of the dragon. Rule6: This is a basic rule: if the beaver swims in the pool next to the house of the mermaid, then the conclusion that \\\"the mermaid refuses to help the liger\\\" follows immediately and effectively. Rule7: If the bear is in Turkey at the moment, then the bear shouts at the bulldog. Rule8: If something does not take over the emperor of the dragonfly and additionally not smile at the shark, then it shouts at the german shepherd. Rule9: The living creature that does not refuse to help the liger will never refuse to help the basenji. Rule10: If there is evidence that one animal, no matter which one, unites with the monkey, then the fangtooth wants to see the mermaid undoubtedly. Rule11: The swan unquestionably captures the king (i.e. the most important piece) of the basenji, in the case where the bulldog brings an oil tank for the swan. Rule12: Here is an important piece of information about the goat: if it has more money than the dragonfly then it does not swear to the woodpecker for sure. Rule13: If something takes over the emperor of the wolf, then it smiles at the shark, too. Rule14: One of the rules of the game is that if the finch captures the king of the gorilla, then the gorilla will, without hesitation, unite with the monkey. Rule15: Regarding the goat, if it is in South America at the moment, then we can conclude that it does not swear to the woodpecker. Rule16: One of the rules of the game is that if the ant does not take over the emperor of the goat, then the goat will never enjoy the companionship of the dragon. Rule17: Regarding the finch, if it has difficulty to find food, then we can conclude that it captures the king (i.e. the most important piece) of the gorilla. Rule18: The basenji will not smile at the shark, in the case where the mermaid does not refuse to help the basenji. Rule19: For the basenji, if the belief is that the swan captures the king (i.e. the most important piece) of the basenji and the rhino reveals a secret to the basenji, then you can add that \\\"the basenji is not going to neglect the cobra\\\" to your conclusions. Rule20: If you are positive that you saw one of the animals neglects the cobra, you can be certain that it will also borrow one of the weapons of the seal. Rule21: There exists an animal which enjoys the companionship of the dragon? Then the frog definitely dances with the basenji. Rule22: The bulldog unquestionably brings an oil tank for the swan, in the case where the bear shouts at the bulldog. Rule23: This is a basic rule: if the frog dances with the basenji, then the conclusion that \\\"the basenji neglects the cobra\\\" follows immediately and effectively. Rule16 is preferred over Rule5. Rule18 is preferred over Rule13. Rule23 is preferred over Rule19. Rule6 is preferred over Rule1. A rule is only applicable if all of its antecedents can be proved. If a rule is preferred over the other, it means whenever both of them can be applied to derive new conclusions and those conclusions contradict with each other (e.g., from one we derive X and from the other we derive not X), we should go with the conclusion from the rule with higher preference. Based on the facts, rules, and preferences, what is the truth value of the statement,  does the basenji shout at the german shepherd? Answer 'proved' if it can be proved, 'disproved' if it can be disproved, and 'unknown' if it can neither be proved nor disproved.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}