{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9b83e461-c565-5407-bc4a-b4f7837a2cb6","task_key":"bbeh~5fobject~5fproperties--full--9b83e461-c565-5407-bc4a-b4f7837a2cb6","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 33 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 33 weird items that went through a few changes. Initially, I had a small Canadian umbrella made of concrete with a smell of garlic, an extra-large Spanish saw made of glass with a smell of wet dog, a small Polish shoe made of plastic with a smell of vinegar, an extra-large Japanese guitar made of ceramic with a smell of gasoline, an extra-extra-small Turkish hat made of ceramic with a smell of gasoline, an extra-extra-large Turkish stapler made of concrete with a smell of garlic, a medium Canadian book made of concrete with a smell of pine needles, a large Chinese wallet made of steel with a smell of wet dog, a large Turkish trash can made of glass with a smell of baking bread, an extra-large Spanish guitar made of concrete with a smell of rose, a large Iranian box made of concrete with a smell of freshly cut grass, an extra-extra-large Polish house made of plastic with a smell of vanilla, an extra-large Chinese candle made of ceramic with a smell of chocolate, an extra-extra-large Canadian clock made of ceramic with a smell of gasoline, a small Iranian piano made of ceramic with a smell of citrus fruits, an extra-small French ruler made of concrete with a smell of burning wood, a medium Afghan clock made of plastic with a smell of coconut, a small Mexican wrench made of ceramic with a smell of wet dog, a small Brazilian trash can made of ceramic with a smell of gasoline, an extra-extra-large Portuguese guitar made of ceramic with a smell of vinegar, a medium British banana made of ceramic with a smell of citrus fruits, a large Italian ring made of steel with a smell of vanilla, a medium French brush made of glass with a smell of citrus fruits, an extra-extra-small Spanish bottle made of glass with a smell of baking bread, an extra-small Russian screwdriver made of concrete with a smell of garlic, an extra-large German guitar made of ceramic with a smell of rose, an extra-large Japanese wrench made of ceramic with a smell of coconut, an extra-extra-large French earring made of concrete with a smell of burning wood, a medium Polish hat made of glass with a smell of garlic, an extra-extra-large Iranian pencil made of plastic with a smell of garlic, a small Iranian vase made of ceramic with a smell of burning wood, an extra-small French cup made of ceramic with a smell of citrus fruits, an extra-large Iranian piano made of ceramic with a smell of vanilla. The color of the items was respectively as follows: the first 1 was beige, the next 2 were black, the next 2 were blue, the next 1 was brown, the next 4 were crimson, the next 1 was cyan, the next 1 was gold, the next 4 were gray, the next 2 were indigo, the next 1 was ivory, the next 1 was khaki, the next 1 was magenta, the next 2 were pink, the next 3 were purple, the next 4 were teal, the next 2 were turquoise, the next 1 was white. Then, if I had an even number of an item type in my collection (e.g. two knives, regardless of their size, color, origin, material, or smell), I threw all of them away. Then, for any item of color beige in my new collection, my mom gave me another one of the same item, but with a gold color, British origin, medium size, vinegar smell, and made of glass. Then, my teacher took any item from my new collection whose color started with \\\"b\\\" or whose name had 6 characters and threw it away. Then, for any item made of plastic in my new collection, my sister changed their color to her favorite color and changed their smell to leather. After this, my new collection had: 1 beige item(s), 1 crimson item(s), 1 cyan item(s), 1 gold item(s), 1 gray item(s), 2 pink item(s), 1 purple item(s), 1 teal item(s), 2 turquoise item(s). Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the extra-extra-large items in my collection.  In my current collection, how many items have the following attributes: color is not cyan, size is not extra-extra-large, material is not steel, smell is not citrus fruits, and origin is not Canadian? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}