{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a5dfb822-5c9d-59b7-9acd-c7f606878cd1","task_key":"bbeh~5fobject~5fproperties--full--a5dfb822-5c9d-59b7-9acd-c7f606878cd1","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 45 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 45 weird items that went through a few changes. Initially, I had a medium Turkish key made of plastic with a smell of citrus fruits, a medium Japanese candle made of concrete with a smell of burning wood, an extra-extra-large Russian piano made of ceramic with a smell of popcorn, a medium Japanese scissors made of steel with a smell of popcorn, an extra-small British jar made of glass with a smell of citrus fruits, a medium Italian fork made of steel with a smell of rose, a small German banana made of plastic with a smell of vinegar, a small Italian bottle made of glass with a smell of vanilla, a large Portuguese candle made of plastic with a smell of vanilla, a large Brazilian speaker made of steel with a smell of chocolate, an extra-extra-large Iranian cup made of plastic with a smell of popcorn, an extra-small Italian bowl made of concrete with a smell of rose, an extra-small German watch made of glass with a smell of chocolate, an extra-extra-small Spanish sunglasses made of concrete with a smell of leather, a small Japanese banana made of ceramic with a smell of leather, an extra-large Portuguese pen made of ceramic with a smell of popcorn, a medium Brazilian cup made of concrete with a smell of rose, a medium Brazilian hat made of ceramic with a smell of garlic, an extra-extra-large Polish drill made of plastic with a smell of coconut, a large Spanish ring made of glass with a smell of coffee, an extra-large German ball made of glass with a smell of pine needles, an extra-large Spanish bowl made of glass with a smell of gasoline, an extra-large Afghan necklace made of glass with a smell of coffee, a small Afghan bird made of concrete with a smell of burning wood, an extra-extra-large Russian car made of plastic with a smell of wet dog, an extra-extra-small French spoon made of ceramic with a smell of vinegar, an extra-extra-large German scissors made of glass with a smell of baking bread, an extra-large Chinese apple made of ceramic with a smell of pine needles, a large Canadian book made of glass with a smell of chocolate, a small Chinese marker made of glass with a smell of citrus fruits, a small Italian ball made of glass with a smell of pine needles, an extra-extra-small German earring made of plastic with a smell of wet dog, a small Turkish wrench made of glass with a smell of vanilla, an extra-extra-small Portuguese calculator made of plastic with a smell of leather, a large Canadian cup made of steel with a smell of chocolate, an extra-extra-small French umbrella made of concrete with a smell of citrus fruits, an extra-extra-large Iranian earring made of glass with a smell of citrus fruits, an extra-extra-small Chinese bird made of plastic with a smell of vinegar, a small Italian watch made of glass with a smell of vinegar, an extra-small Mexican umbrella made of plastic with a smell of chocolate, a medium Brazilian watch made of concrete with a smell of vanilla, an extra-small Turkish clock made of glass with a smell of gasoline, a large Iranian wallet made of steel with a smell of garlic, a medium Chinese clock made of glass with a smell of freshly cut grass, an extra-extra-small Brazilian marker made of concrete with a smell of burning wood. The color of the items was respectively as follows: the first 1 was beige, the next 3 were black, the next 1 was brown, the next 2 were cyan, the next 2 were gold, the next 2 were gray, the next 3 were green, the next 1 was indigo, the next 2 were ivory, the next 2 were khaki, the next 1 was magenta, the next 1 was maroon, the next 3 were orange, the next 2 were pink, the next 2 were purple, the next 5 were red, the next 2 were teal, the next 5 were turquoise, the next 1 was violet, the next 2 were white, the next 2 were yellow. Then, my friend took any item with a smell of vinegar, vanilla or rose in my collection and threw it away. Then, for any item made of glass in my new collection, my sister changed their color to her favorite color and changed their smell to freshly cut grass. After this, my new collection had: 1 beige item(s), 3 black item(s), 13 crimson item(s), 2 gray item(s), 1 green item(s), 1 indigo item(s), 1 ivory item(s), 2 khaki item(s), 1 orange item(s), 1 pink item(s), 1 purple item(s), 1 red item(s), 2 teal item(s), 2 turquoise item(s), 1 white item(s), 1 yellow item(s). Then, my teacher took any item from my new collection whose color started with \\\"b\\\" or whose name had 6 characters and threw it away. Then, for any item of color green in my new collection, my mom gave me another one of the same item, but with a orange color, Chinese origin, small size, leather smell, and made of glass. Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the extra-small items in my collection.  In my current collection, how many items have the following attributes: color is not ivory, size is not extra-small, material is not glass, smell is not citrus fruits, and origin is not Italian? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}