{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f3861c0e-b66b-5415-bf4c-7d2d3dc43185","task_key":"bbeh~5fobject~5fproperties--full--f3861c0e-b66b-5415-bf4c-7d2d3dc43185","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 34 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 34 weird items that went through a few changes. Initially, I had an extra-extra-small Spanish wrench made of ceramic with a smell of gasoline, an extra-extra-small Polish bed made of concrete with a smell of rose, an extra-large Polish canvas made of ceramic with a smell of gasoline, an extra-large Portuguese key made of concrete with a smell of freshly cut grass, an extra-large Turkish piano made of glass with a smell of gasoline, an extra-large German piano made of steel with a smell of vinegar, an extra-extra-large British canvas made of ceramic with a smell of gasoline, an extra-extra-small Italian brush made of plastic with a smell of lavender, an extra-extra-large Brazilian earring made of glass with a smell of citrus fruits, a small Portuguese key made of glass with a smell of gasoline, an extra-extra-small Spanish chair made of concrete with a smell of burning wood, an extra-extra-small Iranian book made of steel with a smell of burning wood, a small Afghan plate made of steel with a smell of chocolate, an extra-extra-large Japanese trash can made of ceramic with a smell of vinegar, a small German screwdriver made of steel with a smell of coffee, an extra-large Iranian shoe made of plastic with a smell of leather, a small British guitar made of glass with a smell of garlic, a large Mexican stapler made of ceramic with a smell of baking bread, an extra-extra-large Brazilian speaker made of plastic with a smell of citrus fruits, a large Turkish apple made of concrete with a smell of burning wood, an extra-large French umbrella made of steel with a smell of citrus fruits, a small Russian spoon made of ceramic with a smell of garlic, an extra-small Canadian spoon made of steel with a smell of coconut, a large Italian plate made of steel with a smell of freshly cut grass, a large Russian ball made of glass with a smell of coffee, a small Japanese plier made of ceramic with a smell of gasoline, an extra-extra-large Spanish candle made of ceramic with a smell of lavender, a large Mexican necklace made of steel with a smell of vanilla, a large Mexican house made of concrete with a smell of garlic, an extra-extra-small Spanish wallet made of ceramic with a smell of rose, a large Brazilian stapler made of ceramic with a smell of citrus fruits, a small Spanish hat made of ceramic with a smell of coconut, an extra-large Japanese house made of steel with a smell of wet dog, an extra-extra-small Russian flower pot made of steel with a smell of popcorn. The color of the items was respectively as follows: the first 1 was beige, the next 1 was black, the next 2 were blue, the next 3 were crimson, the next 1 was gold, the next 2 were indigo, the next 2 were ivory, the next 1 was khaki, the next 4 were maroon, the next 1 was orange, the next 4 were pink, the next 3 were purple, the next 1 was red, the next 2 were teal, the next 1 was turquoise, the next 4 were violet, the next 1 was yellow. Then, my uncle threw away any Spanish item in my collection, but gave me two of the same item, one Iranian and one Polish (all other properties the same). Then, for any item with a smell of vanilla, rose, or burning wood in my new collection, my cousin gifted me another one of that item with the same properties but with her favorite smell (different from the three smells above). After this, my new collection had: 7 item(s) with gasoline smell, 3 item(s) with rose smell, 9 item(s) with chocolate smell, 2 item(s) with freshly cut grass smell, 2 item(s) with vinegar smell, 3 item(s) with lavender smell, 4 item(s) with citrus fruits smell, 4 item(s) with burning wood smell, 2 item(s) with coffee smell, 1 item(s) with leather smell, 3 item(s) with garlic smell, 1 item(s) with baking bread smell, 3 item(s) with coconut smell, 1 item(s) with vanilla smell, 1 item(s) with wet dog smell, 1 item(s) with popcorn smell. Then, if I had an even number of an item type in my new collection (e.g. two knives, regardless of their size, color, origin, material, or smell), I threw all of them away. Then, my friend took any item with a smell of chocolate, wet dog or citrus fruits in my new collection and threw it away. Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the extra-extra-small items in my collection.  In my current collection, how many items have the following attributes: color is not gold, size is not extra-extra-small, material is not plastic, smell is not garlic, and origin is not Russian? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}