{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"15627f2f-eb4d-5111-a4aa-31143049ea18","task_key":"bbeh~5fobject~5fproperties--full--15627f2f-eb4d-5111-a4aa-31143049ea18","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 43 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 43 weird items that went through a few changes. Initially, I had an extra-small Afghan umbrella made of steel with a smell of baking bread, a small French canvas made of ceramic with a smell of vinegar, a medium Turkish ball made of glass with a smell of wet dog, an extra-small Afghan calculator made of ceramic with a smell of lavender, an extra-large Italian piano made of concrete with a smell of pine needles, an extra-small Mexican canvas made of plastic with a smell of citrus fruits, a large Canadian saw made of concrete with a smell of pine needles, an extra-small Portuguese plate made of glass with a smell of wet dog, a medium Japanese table made of concrete with a smell of gasoline, a large British stapler made of concrete with a smell of lavender, an extra-extra-large Turkish house made of plastic with a smell of burning wood, an extra-extra-small Portuguese car made of glass with a smell of baking bread, an extra-small Brazilian clock made of concrete with a smell of popcorn, an extra-extra-small Italian necklace made of ceramic with a smell of lavender, an extra-extra-small Polish box made of steel with a smell of garlic, an extra-small Iranian pencil made of steel with a smell of chocolate, an extra-extra-small British canvas made of plastic with a smell of vanilla, an extra-large Canadian shoe made of ceramic with a smell of wet dog, a large Turkish pen made of glass with a smell of lavender, a large Turkish pen made of steel with a smell of burning wood, an extra-large Canadian wrench made of plastic with a smell of garlic, an extra-extra-large Turkish chair made of ceramic with a smell of gasoline, an extra-extra-large Spanish car made of plastic with a smell of baking bread, an extra-extra-large German table made of ceramic with a smell of rose, an extra-large Iranian bag made of glass with a smell of pine needles, a medium Russian earring made of plastic with a smell of pine needles, an extra-large Chinese box made of concrete with a smell of lavender, an extra-small Spanish marker made of plastic with a smell of garlic, a large Polish key made of steel with a smell of garlic, an extra-small British clock made of steel with a smell of baking bread, an extra-small British sofa made of glass with a smell of vanilla, an extra-small Canadian bicycle made of ceramic with a smell of leather, an extra-extra-large Russian house made of plastic with a smell of vinegar, an extra-extra-small Russian flower pot made of concrete with a smell of baking bread, a large British knife made of ceramic with a smell of vinegar, an extra-small Canadian shoe made of glass with a smell of chocolate, a small Iranian chair made of concrete with a smell of leather, a large Russian bed made of steel with a smell of coconut, an extra-extra-large Italian dog made of steel with a smell of burning wood, a medium Russian sofa made of concrete with a smell of coconut, an extra-large Mexican bird made of ceramic with a smell of rose, an extra-large Italian trash can made of concrete with a smell of vanilla, an extra-large Portuguese pencil made of steel with a smell of chocolate. The color of the items was respectively as follows: the first 4 were black, the next 2 were blue, the next 3 were brown, the next 1 was crimson, the next 3 were cyan, the next 3 were gold, the next 3 were gray, the next 2 were indigo, the next 1 was ivory, the next 2 were khaki, the next 1 was magenta, the next 2 were maroon, the next 2 were orange, the next 1 was pink, the next 1 was purple, the next 1 was silver, the next 1 was teal, the next 3 were turquoise, the next 3 were violet, the next 3 were white, the next 1 was yellow. Then, my friend took any item with a smell of citrus fruits, vanilla or popcorn in my collection and threw it away. Then, my uncle threw away any Chinese item in my new collection, but gave me two of the same item, one Mexican and one Italian (all other properties the same). Then my dad threw away all item of a certain color from my new collection. After this, my new collection only had: 4 black item(s), 1 blue item(s), 3 brown item(s), 1 crimson item(s), 2 cyan item(s), 3 gold item(s), 2 gray item(s), 2 indigo item(s), 1 ivory item(s), 2 khaki item(s), 1 magenta item(s), 3 maroon item(s), 2 orange item(s), 1 pink item(s), 1 teal item(s), 3 turquoise item(s), 3 violet item(s), 2 white item(s), 1 yellow item(s). Then, my teacher took any item from my new collection whose color started with \\\"b\\\" or whose name had 6 characters and threw it away. Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the extra-extra-large items in my collection.  In my current collection, how many items have the following attributes: color is not pink, size is not extra-extra-large, material is not glass, smell is not garlic, and origin is not Polish? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}