{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c15c7aa1-67d8-55d1-9e4f-8dbfb1426366","task_key":"bbeh~5fobject~5fproperties--full--c15c7aa1-67d8-55d1-9e4f-8dbfb1426366","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 30 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 30 weird items that went through a few changes. Initially, I had an extra-extra-small German plate made of ceramic with a smell of gasoline, a medium Japanese plier made of concrete with a smell of pine needles, an extra-extra-small British screwdriver made of ceramic with a smell of rose, an extra-large Spanish trash can made of plastic with a smell of coconut, a large German trash can made of ceramic with a smell of vinegar, an extra-extra-large Spanish shirt made of glass with a smell of pine needles, a medium Portuguese vase made of plastic with a smell of pine needles, a large Polish plate made of steel with a smell of vinegar, a small Polish speaker made of steel with a smell of coconut, a large Brazilian necklace made of plastic with a smell of popcorn, an extra-large Chinese house made of glass with a smell of chocolate, a small Italian plier made of ceramic with a smell of pine needles, an extra-large Mexican chair made of plastic with a smell of lavender, a medium Canadian calculator made of concrete with a smell of coconut, an extra-extra-small British banana made of concrete with a smell of baking bread, an extra-small Spanish canvas made of steel with a smell of garlic, an extra-large Brazilian stapler made of steel with a smell of leather, a small Afghan jar made of ceramic with a smell of baking bread, a medium Turkish speaker made of ceramic with a smell of popcorn, a large Mexican banana made of steel with a smell of vinegar, an extra-large Chinese ring made of ceramic with a smell of chocolate, an extra-large Russian key made of plastic with a smell of coffee, a large Portuguese hat made of ceramic with a smell of coconut, an extra-extra-large Turkish necklace made of concrete with a smell of popcorn, an extra-large French piano made of steel with a smell of leather, an extra-extra-small Brazilian stapler made of ceramic with a smell of vanilla, an extra-extra-large Russian bowl made of ceramic with a smell of leather, a small Japanese scissors made of concrete with a smell of leather, a small Portuguese shoe made of steel with a smell of freshly cut grass, a large British bottle made of glass with a smell of wet dog. The color of the items was respectively as follows: the first 1 was brown, the next 1 was crimson, the next 2 were cyan, the next 4 were gray, the next 3 were green, the next 2 were ivory, the next 4 were khaki, the next 2 were magenta, the next 1 was pink, the next 1 was purple, the next 2 were red, the next 1 was silver, the next 4 were teal, the next 1 was turquoise, the next 1 was violet. Then, my teacher took any item from my collection whose color started with \\\"b\\\" or whose name had 6 characters and threw it away. Then, for any item made of concrete in my new collection, my sister changed their color to her favorite color and changed their smell to chocolate. After this, my new collection had: 4 brown item(s), 2 cyan item(s), 4 gray item(s), 3 green item(s), 2 ivory item(s), 1 khaki item(s), 2 magenta item(s), 1 purple item(s), 2 red item(s), 3 teal item(s), 1 turquoise item(s). Then, for any item of color khaki in my new collection, my mom gave me another one of the same item, but with a gray color, Russian origin, small size, rose smell, and made of glass. Then, my brother replaced any item of size extra-extra-small in my new collection with an exact copy but with one specific size (different from extra-extra-small) and another exact copy but with color maroon. After this, my new collection had: 4 medium item(s), 3 extra-extra-large item(s), 6 small item(s), 9 extra-large item(s), 4 large item(s), 2 extra-extra-small item(s). Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the small items in my collection.  In my current collection, how many items have the following attributes: color is not ivory, size is not small, material is not concrete, smell is not coconut, and origin is not Chinese? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}