{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"46d19c38-8e18-5b13-9ae4-d725e0c44bf3","task_key":"bbeh~5fobject~5fproperties--full--46d19c38-8e18-5b13-9ae4-d725e0c44bf3","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 31 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 31 weird items that went through a few changes. Initially, I had a medium Iranian wallet made of steel with a smell of coffee, a small Brazilian fork made of steel with a smell of rose, an extra-small Russian banana made of plastic with a smell of pine needles, an extra-extra-small Italian ruler made of glass with a smell of lavender, an extra-large Canadian stapler made of glass with a smell of leather, an extra-extra-small Brazilian clock made of concrete with a smell of citrus fruits, a large Russian shirt made of concrete with a smell of citrus fruits, an extra-small Brazilian wallet made of ceramic with a smell of citrus fruits, an extra-extra-large Canadian box made of glass with a smell of vanilla, an extra-small British flower pot made of glass with a smell of citrus fruits, a small Spanish guitar made of steel with a smell of chocolate, a small Mexican hammer made of concrete with a smell of gasoline, an extra-large Chinese apple made of glass with a smell of coffee, a small Mexican wallet made of concrete with a smell of wet dog, a large Japanese stapler made of plastic with a smell of chocolate, an extra-extra-large Iranian cup made of glass with a smell of coconut, a medium Spanish dog made of glass with a smell of garlic, a large Polish house made of ceramic with a smell of citrus fruits, an extra-extra-small Mexican jar made of steel with a smell of coconut, an extra-small Russian bowl made of concrete with a smell of burning wood, a small Japanese wrench made of ceramic with a smell of freshly cut grass, an extra-large French table made of steel with a smell of wet dog, a medium Russian jar made of glass with a smell of vinegar, a medium British camera made of glass with a smell of garlic, a medium Spanish trash can made of ceramic with a smell of vanilla, an extra-small Polish chair made of steel with a smell of burning wood, an extra-extra-large Spanish umbrella made of glass with a smell of vanilla, a large Brazilian guitar made of plastic with a smell of garlic, an extra-extra-small British clock made of steel with a smell of vanilla, a medium Spanish shirt made of steel with a smell of rose, an extra-small Afghan fork made of ceramic with a smell of coconut. The color of the items was respectively as follows: the first 1 was beige, the next 2 were black, the next 2 were blue, the next 1 was crimson, the next 2 were gold, the next 1 was gray, the next 1 was indigo, the next 5 were ivory, the next 1 was khaki, the next 2 were magenta, the next 2 were pink, the next 1 was purple, the next 1 was red, the next 3 were silver, the next 2 were teal, the next 2 were violet, the next 2 were yellow. Then, my teacher took any item from my collection whose color started with \\\"b\\\" or whose name had 6 characters and threw it away. Then, for any item with a smell of vanilla, leather, or lavender in my new collection, my cousin gifted me another one of that item with the same properties but with her favorite smell (different from the three smells above). After this, my new collection had: 4 item(s) with citrus fruits smell, 4 item(s) with vanilla smell, 7 item(s) with coconut smell, 1 item(s) with coffee smell, 1 item(s) with chocolate smell, 1 item(s) with garlic smell, 2 item(s) with burning wood smell, 1 item(s) with wet dog smell, 1 item(s) with vinegar smell, 1 item(s) with rose smell. Then, my friend took any item with a smell of lavender, coffee or vinegar in my new collection and threw it away. Then my dad threw away all item of a certain color from my new collection. After this, my new collection only had: 1 crimson item(s), 1 gold item(s), 2 gray item(s), 1 indigo item(s), 1 ivory item(s), 1 khaki item(s), 2 magenta item(s), 2 pink item(s), 1 red item(s), 3 teal item(s), 2 violet item(s), 2 yellow item(s). Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the extra-extra-large items in my collection.  In my current collection, how many items have the following attributes: color is not ivory, size is not extra-extra-large, material is not glass, smell is not citrus fruits, and origin is not Mexican? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}