{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d548bf83-ba01-5f33-955c-608925df67ed","task_key":"bbeh~5fboardgame~5fqa--full--d548bf83-ba01-5f33-955c-608925df67ed","task_revision_id":"1","upstream_id":"","short_description":"A few players are playing a boardgame. The current state of the game is as…","config":"bbeh_boardgame_qa","split":"full","body":"{\"input\":\"A few players are playing a boardgame. The current state of the game is as follows. The badger builds a power plant near the green fields of the finch. The basenji suspects the truthfulness of the flamingo. The bison shouts at the dragon. The seahorse has a basketball with a diameter of 23 inches, and is watching a movie from 2014. The seahorse manages to convince the songbird. The seal acquires a photograph of the flamingo. And the rules of the game are as follows. Rule1: If at least one animal disarms the llama, then the german shepherd does not surrender to the dolphin. Rule2: The living creature that falls on a square of the shark will also suspect the truthfulness of the german shepherd, without a doubt. Rule3: One of the rules of the game is that if the beetle does not pay some $$$ to the beaver, then the beaver will never smile at the rhino. Rule4: The seahorse will disarm the cobra if it (the seahorse) is watching a movie that was released before Facebook was founded. Rule5: If the cobra suspects the truthfulness of the german shepherd, then the german shepherd surrenders to the dolphin. Rule6: If something builds a power plant near the green fields of the cougar, then it does not swim inside the pool located besides the house of the crow. Rule7: One of the rules of the game is that if the wolf disarms the beaver, then the beaver will, without hesitation, smile at the rhino. Rule8: There exists an animal which swims in the pool next to the house of the crow? Then the cobra definitely falls on a square of the shark. Rule9: If you see that something suspects the truthfulness of the gadwall and dances with the crow, what can you certainly conclude? You can conclude that it does not negotiate a deal with the otter. Rule10: One of the rules of the game is that if the basenji suspects the truthfulness of the flamingo, then the flamingo will, without hesitation, stop the victory of the worm. Rule11: This is a basic rule: if the dinosaur brings an oil tank for the german shepherd, then the conclusion that \\\"the german shepherd dances with the crow\\\" follows immediately and effectively. Rule12: If at least one animal stops the victory of the worm, then the wolf disarms the beaver. Rule13: Here is an important piece of information about the seahorse: if it has a basketball that fits in a 32.8 x 24.8 x 29.1 inches box then it disarms the cobra for sure. Rule14: If you are positive that you saw one of the animals shouts at the dragon, you can be certain that it will also swim in the pool next to the house of the crow. Rule15: The akita enjoys the company of the cobra whenever at least one animal builds a power plant close to the green fields of the finch. Rule16: If something surrenders to the dolphin, then it suspects the truthfulness of the gadwall, too. Rule17: The dinosaur brings an oil tank for the german shepherd whenever at least one animal smiles at the rhino. Rule1 is preferred over Rule5. Rule3 is preferred over Rule7. Rule6 is preferred over Rule14. A rule is only applicable if all of its antecedents can be proved. If a rule is preferred over the other, it means whenever both of them can be applied to derive new conclusions and those conclusions contradict with each other (e.g., from one we derive X and from the other we derive not X), we should go with the conclusion from the rule with higher preference. Based on the facts, rules, and preferences, what is the truth value of the statement,  does the german shepherd negotiate a deal with the otter? Answer 'proved' if it can be proved, 'disproved' if it can be disproved, and 'unknown' if it can neither be proved nor disproved.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}