{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7d5dd243-ff13-59ff-9a0b-f7a8f9f74521","task_key":"bbeh~5fboardgame~5fqa--full--7d5dd243-ff13-59ff-9a0b-f7a8f9f74521","task_revision_id":"1","upstream_id":"","short_description":"A few players are playing a boardgame. The current state of the game is as…","config":"bbeh_boardgame_qa","split":"full","body":"{\"input\":\"A few players are playing a boardgame. The current state of the game is as follows. The dugong creates one castle for the coyote, and disarms the dragonfly. The finch has a cappuccino. And the rules of the game are as follows. Rule1: The living creature that acquires a photo of the seal will also reveal something that is supposed to be a secret to the goat, without a doubt. Rule2: For the goat, if you have two pieces of evidence 1) the dugong reveals a secret to the goat and 2) the fish does not suspect the truthfulness of the goat, then you can add that the goat will never refuse to help the beaver to your conclusions. Rule3: The vampire does not create a castle for the goat, in the case where the finch stops the victory of the vampire. Rule4: If something does not stop the victory of the lizard, then it does not unite with the starling. Rule5: If you are positive that you saw one of the animals refuses to help the monkey, you can be certain that it will also invest in the company whose owner is the woodpecker. Rule6: If you see that something creates one castle for the coyote and disarms the dragonfly, what can you certainly conclude? You can conclude that it also acquires a photograph of the seal. Rule7: One of the rules of the game is that if the beaver swims inside the pool located besides the house of the bee, then the bee will, without hesitation, refuse to help the monkey. Rule8: If something does not swear to the lizard, then it creates one castle for the goat. Rule9: The poodle unites with the starling whenever at least one animal invests in the company whose owner is the woodpecker. Rule10: There exists an animal which swears to the dalmatian? Then, the bee definitely does not refuse to help the monkey. Rule11: The living creature that refuses to help the beaver will also enjoy the company of the peafowl, without a doubt. Rule12: If at least one animal enjoys the company of the peafowl, then the beaver swims inside the pool located besides the house of the bee. Rule13: If the finch has something to drink, then the finch stops the victory of the vampire. Rule14: One of the rules of the game is that if the vampire does not create one castle for the goat, then the goat will, without hesitation, refuse to help the beaver. Rule15: There exists an animal which builds a power plant near the green fields of the german shepherd? Then, the bee definitely does not invest in the company owned by the woodpecker. Rule10 is preferred over Rule7. Rule15 is preferred over Rule5. Rule2 is preferred over Rule14. Rule4 is preferred over Rule9. Rule8 is preferred over Rule3. A rule is only applicable if all of its antecedents can be proved. If a rule is preferred over the other, it means whenever both of them can be applied to derive new conclusions and those conclusions contradict with each other (e.g., from one we derive X and from the other we derive not X), we should go with the conclusion from the rule with higher preference. Based on the facts, rules, and preferences, what is the truth value of the statement,  does the poodle unite with the starling? Answer 'proved' if it can be proved, 'disproved' if it can be disproved, and 'unknown' if it can neither be proved nor disproved.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}