{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"862748cd-98a6-530b-bca4-947d4c5869f9","task_key":"bbeh~5fobject~5fproperties--full--862748cd-98a6-530b-bca4-947d4c5869f9","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 41 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 41 weird items that went through a few changes. Initially, I had an extra-extra-large Turkish knife made of plastic with a smell of popcorn, a large Iranian candle made of ceramic with a smell of wet dog, an extra-extra-small Afghan stapler made of concrete with a smell of vanilla, an extra-small Canadian table made of steel with a smell of vanilla, a small Mexican house made of plastic with a smell of lavender, a small Chinese jar made of steel with a smell of chocolate, a small Brazilian guitar made of plastic with a smell of rose, a large Polish guitar made of plastic with a smell of chocolate, a medium Canadian spoon made of ceramic with a smell of popcorn, an extra-small Turkish ruler made of ceramic with a smell of gasoline, an extra-extra-large Portuguese book made of ceramic with a smell of gasoline, a medium British stapler made of plastic with a smell of leather, a small Brazilian clock made of plastic with a smell of baking bread, an extra-small Afghan marker made of plastic with a smell of garlic, a small Japanese screwdriver made of steel with a smell of popcorn, a large Brazilian jar made of glass with a smell of pine needles, a large Turkish dog made of glass with a smell of coconut, a medium Brazilian vase made of steel with a smell of vinegar, an extra-extra-small Turkish bird made of concrete with a smell of rose, a large Spanish spoon made of ceramic with a smell of garlic, an extra-large Japanese house made of glass with a smell of pine needles, an extra-large Spanish table made of ceramic with a smell of lavender, a large Russian piano made of plastic with a smell of citrus fruits, an extra-large Japanese bed made of concrete with a smell of freshly cut grass, an extra-small Chinese book made of steel with a smell of rose, a medium British key made of steel with a smell of vinegar, a large Polish book made of glass with a smell of gasoline, a large Iranian spoon made of plastic with a smell of freshly cut grass, an extra-small German house made of plastic with a smell of garlic, an extra-large Portuguese plier made of concrete with a smell of garlic, an extra-large Iranian bed made of plastic with a smell of freshly cut grass, an extra-small Turkish earring made of plastic with a smell of pine needles, an extra-extra-small Italian earring made of steel with a smell of popcorn, an extra-extra-small Polish spoon made of plastic with a smell of coffee, an extra-extra-small Brazilian key made of glass with a smell of coffee, a large Turkish table made of plastic with a smell of citrus fruits, a large Mexican trash can made of plastic with a smell of lavender, a small Chinese drill made of ceramic with a smell of popcorn, a medium Brazilian calculator made of ceramic with a smell of burning wood, an extra-extra-large Turkish bed made of glass with a smell of burning wood, an extra-large Portuguese box made of ceramic with a smell of coffee. The color of the items was respectively as follows: the first 4 were black, the next 1 was brown, the next 1 was cyan, the next 1 was gold, the next 3 were gray, the next 2 were green, the next 2 were indigo, the next 3 were ivory, the next 1 was khaki, the next 1 was magenta, the next 2 were maroon, the next 2 were orange, the next 2 were pink, the next 3 were purple, the next 4 were red, the next 1 was silver, the next 2 were teal, the next 4 were turquoise, the next 1 was white, the next 1 was yellow. Then, for any item of color gold in my collection, my mom gave me another one of the same item, but with a magenta color, Japanese origin, large size, popcorn smell, and made of plastic. Then, if I had an even number of an item type in my new collection (e.g. two knives, regardless of their size, color, origin, material, or smell), I threw all of them away. Then, my uncle threw away any Afghan item in my new collection, but gave me two of the same item, one German and one Portuguese (all other properties the same). Then my dad threw away all item of a certain color from my new collection. After this, my new collection only had: 3 black item(s), 1 brown item(s), 1 gold item(s), 1 green item(s), 3 indigo item(s), 2 ivory item(s), 1 khaki item(s), 2 magenta item(s), 1 maroon item(s), 2 orange item(s), 2 pink item(s), 1 purple item(s), 3 red item(s), 4 turquoise item(s), 1 white item(s), 1 yellow item(s). Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the small items in my collection.  In my current collection, how many items have the following attributes: color is not black, size is not small, material is not concrete, smell is not freshly cut grass, and origin is not Spanish? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}