{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"892a9082-b3dc-5012-b5a9-26d846459cac","task_key":"bbeh~5fboardgame~5fqa--full--892a9082-b3dc-5012-b5a9-26d846459cac","task_revision_id":"1","upstream_id":"","short_description":"A few players are playing a boardgame. The current state of the game is as…","config":"bbeh_boardgame_qa","split":"full","body":"{\"input\":\"A few players are playing a boardgame. The current state of the game is as follows. The dugong surrenders to the shark. The dugong swears to the goose. The finch has a card that is red in color. The flamingo has 8 friends that are kind and 1 friend that is not, and has a 16 x 20 inches notebook. The lizard swears to the bison. And the rules of the game are as follows. Rule1: In order to conclude that the chihuahua neglects the peafowl, two pieces of evidence are required: firstly the frog should neglect the chihuahua and secondly the poodle should tear down the castle that belongs to the chihuahua. Rule2: If something does not surrender to the cougar, then it neglects the chihuahua. Rule3: The crab unquestionably disarms the cobra, in the case where the flamingo enjoys the company of the crab. Rule4: If you are positive that you saw one of the animals surrenders to the dugong, you can be certain that it will also enjoy the company of the frog. Rule5: If there is evidence that one animal, no matter which one, hides her cards from the bear, then the woodpecker stops the victory of the duck undoubtedly. Rule6: There exists an animal which surrenders to the llama? Then the duck definitely hides her cards from the bear. Rule7: If the swan captures the king of the poodle, then the poodle tears down the castle of the chihuahua. Rule8: Regarding the flamingo, if it has a notebook that fits in a 20.3 x 14.1 inches box, then we can conclude that it enjoys the companionship of the crab. Rule9: If the elk enjoys the companionship of the frog, then the frog is not going to surrender to the cougar. Rule10: The swan does not capture the king of the poodle whenever at least one animal disarms the mouse. Rule11: If the finch has a card whose color appears in the flag of France, then the finch does not stop the victory of the elk. Rule12: The living creature that surrenders to the flamingo will also capture the king (i.e. the most important piece) of the poodle, without a doubt. Rule13: Here is an important piece of information about the flamingo: if it has fewer than twelve friends then it enjoys the company of the crab for sure. Rule14: The living creature that tears down the castle that belongs to the camel will never surrender to the llama. Rule15: This is a basic rule: if the crow builds a power plant close to the green fields of the coyote, then the conclusion that \\\"the coyote builds a power plant near the green fields of the chihuahua\\\" follows immediately and effectively. Rule16: This is a basic rule: if the coyote builds a power plant close to the green fields of the chihuahua, then the conclusion that \\\"the chihuahua swears to the finch\\\" follows immediately and effectively. Rule17: The elk surrenders to the dugong whenever at least one animal disarms the dalmatian. Rule18: If there is evidence that one animal, no matter which one, disarms the cobra, then the swan surrenders to the flamingo undoubtedly. Rule19: The living creature that neglects the peafowl will never fall on a square of the rhino. Rule20: If the lizard swears to the bison, then the bison surrenders to the llama. Rule21: Are you certain that one of the animals swears to the goose and also at the same time surrenders to the shark? Then you can also be certain that the same animal disarms the dalmatian. Rule22: This is a basic rule: if the finch does not stop the victory of the elk, then the conclusion that the elk will not surrender to the dugong follows immediately and effectively. Rule23: If something does not create a castle for the basenji, then it does not hide her cards from the bear. Rule24: There exists an animal which stops the victory of the duck? Then the crow definitely builds a power plant near the green fields of the coyote. Rule10 is preferred over Rule12. Rule14 is preferred over Rule20. Rule17 is preferred over Rule22. Rule23 is preferred over Rule6. A rule is only applicable if all of its antecedents can be proved. If a rule is preferred over the other, it means whenever both of them can be applied to derive new conclusions and those conclusions contradict with each other (e.g., from one we derive X and from the other we derive not X), we should go with the conclusion from the rule with higher preference. Based on the facts, rules, and preferences, what is the truth value of the statement,  does the chihuahua fall on a square of the rhino? Answer 'proved' if it can be proved, 'disproved' if it can be disproved, and 'unknown' if it can neither be proved nor disproved.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}