{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"15536da9-91f9-5eef-95bc-3d56c29369af","task_key":"bbeh~5fboardgame~5fqa--full--15536da9-91f9-5eef-95bc-3d56c29369af","task_revision_id":"1","upstream_id":"","short_description":"A few players are playing a boardgame. The current state of the game is as…","config":"bbeh_boardgame_qa","split":"full","body":"{\"input\":\"A few players are playing a boardgame. The current state of the game is as follows. The bison has 72 dollars. The bison has a basketball with a diameter of 16 inches. The gadwall has 47 dollars. The owl is watching a movie from 1930. The poodle has a cell phone. The fangtooth does not dance with the walrus. And the rules of the game are as follows. Rule1: The bison will not enjoy the companionship of the german shepherd if it (the bison) is in Canada at the moment. Rule2: If you are positive that one of the animals does not negotiate a deal with the walrus, you can be certain that it will pay money to the mannikin without a doubt. Rule3: If something does not capture the king (i.e. the most important piece) of the bulldog, then it unites with the goose. Rule4: If at least one animal enjoys the company of the german shepherd, then the cobra hides the cards that she has from the lizard. Rule5: There exists an animal which creates one castle for the reindeer? Then the mermaid definitely refuses to help the bear. Rule6: This is a basic rule: if the mannikin leaves the houses occupied by the lizard, then the conclusion that \\\"the lizard will not manage to persuade the seahorse\\\" follows immediately and effectively. Rule7: Here is an important piece of information about the bison: if it has more money than the gadwall then it enjoys the companionship of the german shepherd for sure. Rule8: If the flamingo tears down the castle of the mermaid, then the mermaid is not going to refuse to help the bear. Rule9: This is a basic rule: if the fangtooth pays money to the mannikin, then the conclusion that \\\"the mannikin leaves the houses that are occupied by the lizard\\\" follows immediately and effectively. Rule10: This is a basic rule: if the cobra hides her cards from the lizard, then the conclusion that \\\"the lizard hides her cards from the coyote\\\" follows immediately and effectively. Rule11: If there is evidence that one animal, no matter which one, unites with the swallow, then the poodle is not going to create one castle for the reindeer. Rule12: Here is an important piece of information about the owl: if it is watching a movie that was released before world war 2 started then it destroys the wall built by the cobra for sure. Rule13: For the goose, if you have two pieces of evidence 1) the lizard acquires a photo of the goose and 2) the stork unites with the goose, then you can add \\\"goose will never enjoy the companionship of the goat\\\" to your conclusions. Rule14: Regarding the bison, if it has a basketball that fits in a 17.2 x 26.1 x 13.8 inches box, then we can conclude that it does not enjoy the company of the german shepherd. Rule15: One of the rules of the game is that if the duck shouts at the stork, then the stork will, without hesitation, capture the king of the bulldog. Rule16: If there is evidence that one animal, no matter which one, refuses to help the bear, then the stork is not going to capture the king of the bulldog. Rule17: This is a basic rule: if the vampire calls the goose, then the conclusion that \\\"the goose enjoys the company of the goat\\\" follows immediately and effectively. Rule18: Be careful when something hides the cards that she has from the coyote but does not manage to convince the seahorse because in this case it will, surely, acquire a photo of the goose (this may or may not be problematic). Rule19: The poodle will create one castle for the reindeer if it (the poodle) has a device to connect to the internet. Rule20: The goat unquestionably reveals a secret to the akita, in the case where the goose does not enjoy the companionship of the goat. Rule1 is preferred over Rule7. Rule14 is preferred over Rule7. Rule15 is preferred over Rule16. Rule17 is preferred over Rule13. Rule19 is preferred over Rule11. Rule8 is preferred over Rule5. A rule is only applicable if all of its antecedents can be proved. If a rule is preferred over the other, it means whenever both of them can be applied to derive new conclusions and those conclusions contradict with each other (e.g., from one we derive X and from the other we derive not X), we should go with the conclusion from the rule with higher preference. Based on the facts, rules, and preferences, what is the truth value of the statement,  does the goat reveal a secret to the akita? Answer 'proved' if it can be proved, 'disproved' if it can be disproved, and 'unknown' if it can neither be proved nor disproved.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}