{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0e6dea60-bbd0-570d-ac17-5941fc10b27a","task_key":"bbeh~5fobject~5fproperties--full--0e6dea60-bbd0-570d-ac17-5941fc10b27a","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 43 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 43 weird items that went through a few changes. Initially, I had an extra-large British plier made of ceramic with a smell of burning wood, a small Mexican brush made of ceramic with a smell of vanilla, an extra-large French guitar made of steel with a smell of lavender, an extra-extra-large Russian bottle made of concrete with a smell of vinegar, an extra-large Turkish necklace made of concrete with a smell of citrus fruits, a small Afghan canvas made of ceramic with a smell of coconut, an extra-extra-large French bag made of plastic with a smell of wet dog, a small Portuguese saw made of glass with a smell of pine needles, a small Italian vase made of plastic with a smell of wet dog, a medium Portuguese necklace made of plastic with a smell of chocolate, an extra-extra-small Afghan car made of plastic with a smell of lavender, a medium Polish scissors made of concrete with a smell of chocolate, an extra-extra-large Polish plier made of steel with a smell of gasoline, an extra-small Canadian pencil made of plastic with a smell of chocolate, an extra-extra-small Portuguese scissors made of concrete with a smell of coconut, an extra-large Russian umbrella made of glass with a smell of vanilla, an extra-extra-small Spanish apple made of plastic with a smell of leather, an extra-extra-small French fork made of plastic with a smell of garlic, an extra-small Iranian drill made of plastic with a smell of wet dog, a large Japanese flower pot made of plastic with a smell of coconut, an extra-extra-large Canadian earring made of steel with a smell of vanilla, a medium British umbrella made of glass with a smell of chocolate, an extra-large Iranian cup made of ceramic with a smell of citrus fruits, a small Portuguese candle made of steel with a smell of vinegar, an extra-extra-large Turkish banana made of glass with a smell of leather, a medium Polish screwdriver made of ceramic with a smell of lavender, a medium Mexican piano made of concrete with a smell of freshly cut grass, an extra-small Spanish shirt made of concrete with a smell of burning wood, a small Canadian bag made of concrete with a smell of vinegar, a small Canadian bottle made of steel with a smell of gasoline, an extra-extra-large German watch made of glass with a smell of baking bread, a medium British scissors made of glass with a smell of rose, an extra-extra-small Russian car made of plastic with a smell of coconut, an extra-extra-small Afghan bottle made of plastic with a smell of coffee, an extra-large German pen made of steel with a smell of popcorn, a large German bird made of ceramic with a smell of citrus fruits, a medium Portuguese drill made of steel with a smell of wet dog, a small Afghan vase made of steel with a smell of gasoline, an extra-extra-large Mexican ruler made of steel with a smell of coconut, an extra-extra-large British camera made of steel with a smell of gasoline, a small Portuguese bird made of ceramic with a smell of baking bread, an extra-large Iranian ball made of concrete with a smell of gasoline, a medium Brazilian candle made of concrete with a smell of coffee. The color of the items was respectively as follows: the first 3 were beige, the next 1 was black, the next 2 were blue, the next 3 were brown, the next 2 were crimson, the next 4 were cyan, the next 2 were gold, the next 1 was gray, the next 1 was green, the next 1 was indigo, the next 1 was magenta, the next 1 was orange, the next 1 was pink, the next 1 was purple, the next 4 were red, the next 1 was silver, the next 5 were teal, the next 3 were turquoise, the next 3 were violet, the next 3 were yellow. Then, for each item of size medium in my collection, my aunt took it but then gave me another one of the same item made of her favorite material and with a smell of garlic (all other properties the same). After this, my collection had: 6 item(s) made of ceramic, 9 item(s) made of steel, 14 item(s) made of concrete, 10 item(s) made of plastic, 4 item(s) made of glass. Then, my brother replaced any item of size medium in my new collection with an exact copy but with one specific size (different from medium) and another exact copy but with color gold. After this, my new collection had: 7 extra-large item(s), 9 small item(s), 8 extra-extra-large item(s), 14 extra-extra-small item(s), 3 extra-small item(s), 8 medium item(s), 2 large item(s). Then, my teacher took any item from my new collection whose color started with \\\"b\\\" or whose name had 6 characters and threw it away. Then, my friend took any item with a smell of vanilla, burning wood or chocolate in my new collection and threw it away. Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the extra-extra-small items in my collection.  In my current collection, how many items have the following attributes: either color is gold, or size is extra-extra-small, or material is glass, or smell is leather, or origin is German? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}