{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"36556d71-4578-5982-9c65-ece5dbb18c50","task_key":"bbeh~5fobject~5fproperties--full--36556d71-4578-5982-9c65-ece5dbb18c50","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 31 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 31 weird items that went through a few changes. Initially, I had an extra-extra-small Iranian earring made of glass with a smell of citrus fruits, an extra-extra-large Chinese ruler made of glass with a smell of burning wood, an extra-extra-large German flower pot made of steel with a smell of freshly cut grass, a medium Chinese flower pot made of ceramic with a smell of leather, a large Italian knife made of concrete with a smell of rose, an extra-large Japanese candle made of plastic with a smell of pine needles, an extra-large Polish screwdriver made of steel with a smell of burning wood, an extra-large German bowl made of ceramic with a smell of lavender, a medium Spanish house made of concrete with a smell of garlic, an extra-extra-large Afghan piano made of glass with a smell of gasoline, an extra-extra-large British shoe made of glass with a smell of citrus fruits, a medium Portuguese wrench made of ceramic with a smell of gasoline, a medium Polish bed made of concrete with a smell of coconut, a medium Russian house made of concrete with a smell of rose, a medium Japanese knife made of plastic with a smell of popcorn, an extra-small Turkish car made of concrete with a smell of baking bread, an extra-extra-large French bicycle made of ceramic with a smell of coconut, a large British scissors made of steel with a smell of coffee, a medium British scissors made of plastic with a smell of vinegar, an extra-large German apple made of ceramic with a smell of leather, a large Turkish jar made of plastic with a smell of gasoline, a small Japanese ball made of steel with a smell of vanilla, a large Mexican piano made of ceramic with a smell of popcorn, an extra-extra-large Portuguese bag made of concrete with a smell of wet dog, an extra-large Iranian chair made of steel with a smell of vinegar, an extra-extra-large Italian house made of steel with a smell of wet dog, a small Polish car made of ceramic with a smell of citrus fruits, an extra-large Japanese book made of steel with a smell of citrus fruits, a small Turkish trash can made of plastic with a smell of coffee, a medium French dog made of steel with a smell of coffee, an extra-large Afghan umbrella made of ceramic with a smell of coconut. The color of the items was respectively as follows: the first 2 were beige, the next 3 were black, the next 4 were brown, the next 2 were crimson, the next 2 were gold, the next 1 was gray, the next 1 was green, the next 1 was indigo, the next 3 were ivory, the next 1 was khaki, the next 2 were magenta, the next 1 was maroon, the next 2 were orange, the next 1 was pink, the next 1 was red, the next 2 were silver, the next 1 was turquoise, the next 1 was white. Then, my brother replaced any item of size medium in my collection with an exact copy but with one specific size (different from medium) and another exact copy but with color beige. After this, my collection had: 9 extra-extra-small item(s), 7 extra-extra-large item(s), 8 medium item(s), 4 large item(s), 7 extra-large item(s), 1 extra-small item(s), 3 small item(s). Then, for any item made of plastic in my new collection, my sister changed their color to her favorite color and changed their smell to vinegar. After this, my new collection had: 8 beige item(s), 3 black item(s), 3 brown item(s), 2 crimson item(s), 2 gold item(s), 1 gray item(s), 8 indigo item(s), 2 ivory item(s), 1 khaki item(s), 1 magenta item(s), 1 maroon item(s), 2 orange item(s), 1 pink item(s), 1 red item(s), 1 silver item(s), 1 turquoise item(s), 1 white item(s). Then, for each item of size extra-large in my new collection, my aunt took it but then gave me another one of the same item made of her favorite material and with a smell of rose (all other properties the same). After this, my new collection had: 4 item(s) made of glass, 7 item(s) made of ceramic, 9 item(s) made of concrete, 6 item(s) made of steel, 13 item(s) made of plastic. Then, my teacher took any item from my new collection whose color started with \\\"b\\\" or whose name had 6 characters and threw it away. Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the medium items in my collection.  In my current collection, how many items have the following attributes: color is not magenta, size is not medium, material is not concrete, smell is not gasoline, and origin is not Afghan? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}