{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2710c711-221a-5bb9-9b89-052303f6473c","task_key":"bbeh~5fboardgame~5fqa--full--2710c711-221a-5bb9-9b89-052303f6473c","task_revision_id":"1","upstream_id":"","short_description":"A few players are playing a boardgame. The current state of the game is as…","config":"bbeh_boardgame_qa","split":"full","body":"{\"input\":\"A few players are playing a boardgame. The current state of the game is as follows. The cobra has 67 dollars. The songbird has 38 dollars. The vampire has 71 dollars. The vampire has a card that is red in color, and is currently in Kenya. The dugong does not enjoy the company of the akita. And the rules of the game are as follows. Rule1: If you are positive that one of the animals does not enjoy the companionship of the akita, you can be certain that it will reveal a secret to the goat without a doubt. Rule2: If you are positive that you saw one of the animals refuses to help the monkey, you can be certain that it will also invest in the company owned by the woodpecker. Rule3: There exists an animal which swears to the dalmatian? Then, the bee definitely does not refuse to help the monkey. Rule4: One of the rules of the game is that if the vampire does not create a castle for the goat, then the goat will, without hesitation, refuse to help the beaver. Rule5: There exists an animal which enjoys the company of the peafowl? Then the beaver definitely swims in the pool next to the house of the bee. Rule6: If at least one animal builds a power plant near the green fields of the german shepherd, then the bee does not invest in the company owned by the woodpecker. Rule7: One of the rules of the game is that if the beaver swims in the pool next to the house of the bee, then the bee will, without hesitation, refuse to help the monkey. Rule8: If the vampire is in Africa at the moment, then the vampire does not create one castle for the goat. Rule9: The living creature that refuses to help the beaver will also enjoy the company of the peafowl, without a doubt. Rule10: If there is evidence that one animal, no matter which one, invests in the company owned by the woodpecker, then the poodle is not going to unite with the starling. Rule11: For the goat, if you have two pieces of evidence 1) the dugong reveals something that is supposed to be a secret to the goat and 2) the fish does not suspect the truthfulness of the goat, then you can add that the goat will never refuse to help the beaver to your conclusions. Rule12: From observing that an animal does not stop the victory of the lizard, one can conclude that it unites with the starling. Rule11 is preferred over Rule4. Rule12 is preferred over Rule10. Rule3 is preferred over Rule7. Rule6 is preferred over Rule2. A rule is only applicable if all of its antecedents can be proved. If a rule is preferred over the other, it means whenever both of them can be applied to derive new conclusions and those conclusions contradict with each other (e.g., from one we derive X and from the other we derive not X), we should go with the conclusion from the rule with higher preference. Based on the facts, rules, and preferences, what is the truth value of the statement,  does the poodle unite with the starling? Answer 'proved' if it can be proved, 'disproved' if it can be disproved, and 'unknown' if it can neither be proved nor disproved.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}