{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"589afb05-412e-5b82-8114-f416d910a89e","task_key":"bbeh~5fboardgame~5fqa--full--589afb05-412e-5b82-8114-f416d910a89e","task_revision_id":"1","upstream_id":"","short_description":"A few players are playing a boardgame. The current state of the game is as…","config":"bbeh_boardgame_qa","split":"full","body":"{\"input\":\"A few players are playing a boardgame. The current state of the game is as follows. The ant has 62 dollars. The beetle has a football with a radius of 21 inches. The butterfly has a trumpet. The butterfly struggles to find food. The german shepherd has 92 dollars. The german shepherd is named Teddy, and is watching a movie from 2015. The goose invests in the company whose owner is the basenji. The mannikin is named Tarzan. The rhino has 67 dollars. And the rules of the game are as follows. Rule1: If at least one animal hides her cards from the cougar, then the chihuahua suspects the truthfulness of the ant. Rule2: The zebra calls the mermaid whenever at least one animal invests in the company owned by the basenji. Rule3: From observing that one animal manages to convince the mouse, one can conclude that it also hides the cards that she has from the cougar, undoubtedly. Rule4: Be careful when something refuses to help the mouse and also borrows a weapon from the chinchilla because in this case it will surely not hide her cards from the cougar (this may or may not be problematic). Rule5: The bear unquestionably borrows one of the weapons of the chinchilla, in the case where the butterfly falls on a square that belongs to the bear. Rule6: The beetle will smile at the bear if it (the beetle) has a football that fits in a 50.9 x 48.3 x 49.2 inches box. Rule7: Regarding the german shepherd, if it has a name whose first letter is the same as the first letter of the mannikin's name, then we can conclude that it does not neglect the bear. Rule8: If the german shepherd has more money than the rhino and the ant combined, then the german shepherd does not neglect the bear. Rule9: If the butterfly has difficulty to find food, then the butterfly falls on a square that belongs to the bear. Rule10: From observing that an animal swims in the pool next to the house of the monkey, one can conclude the following: that animal does not swear to the llama. Rule11: For the bear, if the belief is that the beetle smiles at the bear and the german shepherd does not neglect the bear, then you can add \\\"the bear manages to convince the mouse\\\" to your conclusions. Rule12: If you are positive that one of the animals does not negotiate a deal with the leopard, you can be certain that it will not borrow one of the weapons of the chinchilla. Rule13: The living creature that swears to the llama will also swim in the pool next to the house of the otter, without a doubt. Rule14: The badger swears to the llama whenever at least one animal suspects the truthfulness of the ant. Rule15: Regarding the butterfly, if it has a device to connect to the internet, then we can conclude that it falls on a square of the bear. Rule10 is preferred over Rule14. Rule12 is preferred over Rule5. Rule4 is preferred over Rule3. A rule is only applicable if all of its antecedents can be proved. If a rule is preferred over the other, it means whenever both of them can be applied to derive new conclusions and those conclusions contradict with each other (e.g., from one we derive X and from the other we derive not X), we should go with the conclusion from the rule with higher preference. Based on the facts, rules, and preferences, what is the truth value of the statement,  does the badger swim in the pool next to the house of the otter? Answer 'proved' if it can be proved, 'disproved' if it can be disproved, and 'unknown' if it can neither be proved nor disproved.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}