{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"58c809ae-39e4-5693-9c8b-c0858b136b5a","task_key":"bbeh~5fobject~5fproperties--full--58c809ae-39e4-5693-9c8b-c0858b136b5a","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 33 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 33 weird items that went through a few changes. Initially, I had a small French wallet made of steel with a smell of rose, a large Russian guitar made of glass with a smell of leather, a small Mexican saw made of glass with a smell of freshly cut grass, an extra-small French jar made of concrete with a smell of pine needles, an extra-small Polish house made of ceramic with a smell of lavender, a small Canadian dog made of concrete with a smell of citrus fruits, a medium Russian shoe made of concrete with a smell of pine needles, a small Chinese canvas made of plastic with a smell of coffee, an extra-large French bird made of plastic with a smell of popcorn, an extra-large German house made of glass with a smell of freshly cut grass, a large Portuguese trash can made of plastic with a smell of popcorn, an extra-large Spanish wallet made of steel with a smell of coconut, an extra-extra-large Canadian apple made of concrete with a smell of freshly cut grass, an extra-extra-small Italian ball made of steel with a smell of garlic, a large Japanese banana made of plastic with a smell of vinegar, an extra-extra-small Italian screwdriver made of steel with a smell of coconut, an extra-extra-small Japanese banana made of plastic with a smell of gasoline, an extra-small Polish banana made of ceramic with a smell of freshly cut grass, an extra-extra-small Japanese ball made of steel with a smell of vanilla, an extra-extra-large Brazilian hat made of ceramic with a smell of pine needles, a small Iranian earring made of steel with a smell of vanilla, an extra-extra-large Italian pencil made of concrete with a smell of popcorn, a medium British apple made of glass with a smell of leather, a small Iranian calculator made of glass with a smell of rose, a large Mexican trash can made of steel with a smell of chocolate, a small British speaker made of plastic with a smell of rose, an extra-extra-large Mexican drill made of plastic with a smell of garlic, a small German car made of concrete with a smell of gasoline, an extra-extra-large Canadian spoon made of concrete with a smell of lavender, a medium German car made of ceramic with a smell of chocolate, an extra-extra-small Russian apple made of plastic with a smell of gasoline, an extra-large French knife made of concrete with a smell of chocolate, an extra-extra-large Iranian saw made of plastic with a smell of coconut. The color of the items was respectively as follows: the first 1 was beige, the next 1 was black, the next 1 was blue, the next 1 was gold, the next 2 were gray, the next 1 was green, the next 4 were indigo, the next 2 were khaki, the next 2 were magenta, the next 3 were orange, the next 1 was red, the next 1 was silver, the next 2 were teal, the next 2 were turquoise, the next 4 were violet, the next 3 were white, the next 2 were yellow. Then, my teacher took any item from my collection whose color started with \\\"b\\\" or whose name had 6 characters and threw it away. Then my dad threw away all item of a certain color from my new collection. After this, my new collection only had: 1 gold item(s), 2 gray item(s), 1 green item(s), 3 indigo item(s), 1 khaki item(s), 1 magenta item(s), 1 orange item(s), 1 red item(s), 1 silver item(s), 2 turquoise item(s), 4 violet item(s), 3 white item(s), 2 yellow item(s). Then, my friend took any item with a smell of leather, baking bread or burning wood in my new collection and threw it away. Then, my brother replaced any item of size extra-large in my new collection with an exact copy but with one specific size (different from extra-large) and another exact copy but with color green. After this, my new collection had: 2 extra-small item(s), 7 small item(s), 2 medium item(s), 3 extra-large item(s), 2 large item(s), 5 extra-extra-large item(s), 4 extra-extra-small item(s). Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the extra-extra-large items in my collection.  In my current collection, how many items have the following attributes: color is not indigo, size is not extra-extra-large, material is not steel, smell is not vanilla, and origin is not Mexican? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}