{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ff3929d1-f05a-5775-b17b-6f1b4cd58d4e","task_key":"bbeh~5fobject~5fproperties--full--ff3929d1-f05a-5775-b17b-6f1b4cd58d4e","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 30 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 30 weird items that went through a few changes. Initially, I had a small Canadian candle made of glass with a smell of rose, an extra-extra-large British watch made of glass with a smell of garlic, a medium Canadian chair made of ceramic with a smell of leather, a large French spoon made of glass with a smell of popcorn, an extra-large Brazilian scissors made of ceramic with a smell of leather, a large Portuguese apple made of glass with a smell of coconut, an extra-large Italian ball made of concrete with a smell of citrus fruits, an extra-extra-small Russian bicycle made of plastic with a smell of baking bread, a medium Japanese flower pot made of ceramic with a smell of garlic, an extra-extra-small Portuguese sunglasses made of concrete with a smell of pine needles, a large German marker made of ceramic with a smell of coffee, a small Spanish spoon made of glass with a smell of burning wood, an extra-small Chinese wallet made of plastic with a smell of leather, an extra-small Chinese ball made of concrete with a smell of baking bread, an extra-extra-small Russian umbrella made of ceramic with a smell of citrus fruits, a large French cup made of plastic with a smell of citrus fruits, a medium Polish shirt made of concrete with a smell of pine needles, a large Canadian shirt made of glass with a smell of coffee, a small Iranian box made of glass with a smell of lavender, a small Brazilian knife made of steel with a smell of burning wood, a large Iranian apple made of plastic with a smell of baking bread, an extra-extra-small Polish umbrella made of steel with a smell of lavender, an extra-small Polish house made of plastic with a smell of wet dog, a large Japanese bowl made of steel with a smell of wet dog, an extra-large Japanese bicycle made of ceramic with a smell of citrus fruits, a small Spanish knife made of steel with a smell of lavender, an extra-large Polish fork made of concrete with a smell of lavender, a small Brazilian car made of concrete with a smell of coconut, an extra-extra-large French guitar made of glass with a smell of lavender, an extra-extra-large French pencil made of steel with a smell of popcorn. The color of the items was respectively as follows: the first 2 were blue, the next 1 was brown, the next 1 was cyan, the next 1 was gray, the next 1 was green, the next 1 was indigo, the next 2 were khaki, the next 2 were magenta, the next 1 was maroon, the next 1 was orange, the next 1 was pink, the next 2 were purple, the next 3 were red, the next 2 were silver, the next 3 were teal, the next 3 were turquoise, the next 3 were violet. Then, my uncle threw away any Iranian item in my collection, but gave me two of the same item, one Brazilian and one Portuguese (all other properties the same). Then, my friend took any item with a smell of coffee, popcorn or baking bread in my new collection and threw it away. Then, if I had an even number of an item type in my new collection (e.g. two knives, regardless of their size, color, origin, material, or smell), I threw all of them away. Then, my brother replaced any item of size medium in my new collection with an exact copy but with one specific size (different from medium) and another exact copy but with color maroon. After this, my new collection had: 6 small item(s), 2 extra-extra-large item(s), 4 extra-large item(s), 3 large item(s), 1 extra-extra-small item(s), 3 medium item(s), 2 extra-small item(s). Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the extra-extra-large items in my collection.  In my current collection, how many items have the following attributes: color is not magenta, size is not extra-extra-large, material is not glass, smell is not leather, and origin is not Spanish? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}