{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0ad9f9ae-db46-5abc-93da-cdcb9179cf82","task_key":"bbeh~5fboardgame~5fqa--full--0ad9f9ae-db46-5abc-93da-cdcb9179cf82","task_revision_id":"1","upstream_id":"","short_description":"A few players are playing a boardgame. The current state of the game is as…","config":"bbeh_boardgame_qa","split":"full","body":"{\"input\":\"A few players are playing a boardgame. The current state of the game is as follows. The coyote is a public relations specialist. The dove takes over the emperor of the finch. The frog is currently in Colombia. The poodle has a basketball with a diameter of 18 inches, and is currently in Toronto. The cougar does not want to see the frog. And the rules of the game are as follows. Rule1: Regarding the frog, if it is in South America at the moment, then we can conclude that it hugs the monkey. Rule2: If the gadwall does not smile at the finch, then the finch does not pay money to the lizard. Rule3: The living creature that captures the king (i.e. the most important piece) of the cougar will also pay some $$$ to the gadwall, without a doubt. Rule4: Regarding the coyote, if it works in marketing, then we can conclude that it takes over the emperor of the frog. Rule5: There exists an animal which tears down the castle that belongs to the basenji? Then the ant definitely acquires a photograph of the pelikan. Rule6: This is a basic rule: if the cougar does not want to see the frog, then the conclusion that the frog will not hug the monkey follows immediately and effectively. Rule7: Are you certain that one of the animals pays some $$$ to the lizard and also at the same time swims inside the pool located besides the house of the llama? Then you can also be certain that the same animal does not tear down the castle that belongs to the basenji. Rule8: One of the rules of the game is that if the leopard does not disarm the pigeon, then the pigeon will, without hesitation, neglect the snake. Rule9: Here is an important piece of information about the poodle: if it works in marketing then it does not unite with the basenji for sure. Rule10: The finch pays some $$$ to the lizard whenever at least one animal pays some $$$ to the gadwall. Rule11: If something does not create a castle for the butterfly, then it does not acquire a photo of the pelikan. Rule12: If the dove takes over the emperor of the finch, then the finch smiles at the frog. Rule13: If something does not neglect the snake, then it swims inside the pool located besides the house of the camel. Rule14: If there is evidence that one animal, no matter which one, swims in the pool next to the house of the camel, then the finch tears down the castle that belongs to the basenji undoubtedly. Rule15: Regarding the poodle, if it is in South America at the moment, then we can conclude that it unites with the basenji. Rule16: The pigeon does not neglect the snake whenever at least one animal swears to the rhino. Rule17: This is a basic rule: if the poodle unites with the basenji, then the conclusion that \\\"the basenji swears to the rhino\\\" follows immediately and effectively. Rule18: From observing that one animal hugs the monkey, one can conclude that it also captures the king of the cougar, undoubtedly. Rule19: If the poodle has a basketball that fits in a 25.5 x 20.2 x 27.5 inches box, then the poodle unites with the basenji. Rule1 is preferred over Rule6. Rule11 is preferred over Rule5. Rule2 is preferred over Rule10. Rule7 is preferred over Rule14. Rule8 is preferred over Rule16. Rule9 is preferred over Rule15. Rule9 is preferred over Rule19. A rule is only applicable if all of its antecedents can be proved. If a rule is preferred over the other, it means whenever both of them can be applied to derive new conclusions and those conclusions contradict with each other (e.g., from one we derive X and from the other we derive not X), we should go with the conclusion from the rule with higher preference. Based on the facts, rules, and preferences, what is the truth value of the statement,  does the ant acquire a photograph of the pelikan? Answer 'proved' if it can be proved, 'disproved' if it can be disproved, and 'unknown' if it can neither be proved nor disproved.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}