{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f8457bfe-09e2-516d-a924-5dbc7e292935","task_key":"bbeh~5fboardgame~5fqa--full--f8457bfe-09e2-516d-a924-5dbc7e292935","task_revision_id":"1","upstream_id":"","short_description":"A few players are playing a boardgame. The current state of the game is as…","config":"bbeh_boardgame_qa","split":"full","body":"{\"input\":\"A few players are playing a boardgame. The current state of the game is as follows. The fish is named Teddy. The goat has a cutter, and parked her bike in front of the store. The goat is named Mojo. And the rules of the game are as follows. Rule1: Here is an important piece of information about the goat: if it has a device to connect to the internet then it refuses to help the beaver for sure. Rule2: The living creature that does not stop the victory of the lizard will never unite with the starling. Rule3: If you are positive that you saw one of the animals refuses to help the monkey, you can be certain that it will also invest in the company whose owner is the woodpecker. Rule4: If something refuses to help the beaver, then it enjoys the company of the peafowl, too. Rule5: If at least one animal swears to the dalmatian, then the bee does not refuse to help the monkey. Rule6: Regarding the goat, if it has a name whose first letter is the same as the first letter of the fish's name, then we can conclude that it does not refuse to help the beaver. Rule7: If there is evidence that one animal, no matter which one, builds a power plant near the green fields of the german shepherd, then the bee is not going to invest in the company owned by the woodpecker. Rule8: There exists an animal which invests in the company whose owner is the woodpecker? Then the poodle definitely unites with the starling. Rule9: If there is evidence that one animal, no matter which one, enjoys the company of the peafowl, then the beaver swims inside the pool located besides the house of the bee undoubtedly. Rule10: If the goat took a bike from the store, then the goat refuses to help the beaver. Rule11: Here is an important piece of information about the goat: if it has a card whose color starts with the letter \\\"r\\\" then it does not refuse to help the beaver for sure. Rule12: If the beaver swims in the pool next to the house of the bee, then the bee refuses to help the monkey. Rule11 is preferred over Rule1. Rule11 is preferred over Rule10. Rule2 is preferred over Rule8. Rule5 is preferred over Rule12. Rule6 is preferred over Rule1. Rule6 is preferred over Rule10. Rule7 is preferred over Rule3. A rule is only applicable if all of its antecedents can be proved. If a rule is preferred over the other, it means whenever both of them can be applied to derive new conclusions and those conclusions contradict with each other (e.g., from one we derive X and from the other we derive not X), we should go with the conclusion from the rule with higher preference. Based on the facts, rules, and preferences, what is the truth value of the statement,  does the poodle unite with the starling? Answer 'proved' if it can be proved, 'disproved' if it can be disproved, and 'unknown' if it can neither be proved nor disproved.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}