{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cbf043cf-230d-57f3-9d4a-4bb88f2b9c56","task_key":"bbeh~5fobject~5fproperties--full--cbf043cf-230d-57f3-9d4a-4bb88f2b9c56","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 33 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 33 weird items that went through a few changes. Initially, I had an extra-extra-small Italian table made of steel with a smell of citrus fruits, a small Portuguese earring made of ceramic with a smell of rose, a medium Portuguese bottle made of glass with a smell of garlic, an extra-extra-large Italian drill made of steel with a smell of freshly cut grass, a large Mexican chair made of steel with a smell of pine needles, an extra-large Russian plate made of ceramic with a smell of popcorn, a medium British guitar made of plastic with a smell of chocolate, a large Mexican cup made of ceramic with a smell of baking bread, an extra-large French hat made of concrete with a smell of vanilla, an extra-extra-small German book made of steel with a smell of garlic, an extra-extra-large Chinese hammer made of concrete with a smell of burning wood, an extra-extra-small French speaker made of glass with a smell of gasoline, an extra-extra-large Polish sofa made of ceramic with a smell of vanilla, a medium Brazilian bottle made of glass with a smell of garlic, a small French chair made of steel with a smell of garlic, a large British wallet made of ceramic with a smell of wet dog, an extra-small Brazilian candle made of steel with a smell of chocolate, an extra-extra-large German trash can made of ceramic with a smell of baking bread, an extra-large Afghan flower pot made of concrete with a smell of lavender, an extra-large Polish ring made of glass with a smell of coconut, an extra-large British pen made of ceramic with a smell of garlic, a large Afghan plate made of glass with a smell of citrus fruits, an extra-extra-large Iranian banana made of steel with a smell of burning wood, a small Turkish candle made of concrete with a smell of pine needles, an extra-extra-small Brazilian ruler made of plastic with a smell of rose, a medium British ruler made of ceramic with a smell of vinegar, an extra-small Turkish stapler made of concrete with a smell of popcorn, a small French hat made of concrete with a smell of baking bread, an extra-small Spanish flower pot made of glass with a smell of popcorn, an extra-small British pen made of steel with a smell of leather, an extra-extra-large Turkish chair made of steel with a smell of citrus fruits, an extra-extra-small Spanish banana made of ceramic with a smell of pine needles, a small Spanish wrench made of plastic with a smell of gasoline. The color of the items was respectively as follows: the first 1 was beige, the next 2 were black, the next 1 was brown, the next 3 were crimson, the next 1 was gold, the next 1 was gray, the next 2 were green, the next 2 were indigo, the next 4 were khaki, the next 1 was magenta, the next 2 were maroon, the next 1 was orange, the next 3 were pink, the next 3 were purple, the next 1 was red, the next 1 was silver, the next 2 were teal, the next 1 was turquoise, the next 1 was yellow. Then, for each item of size medium in my collection, my aunt took it but then gave me another one of the same item made of her favorite material and with a smell of coconut (all other properties the same). After this, my collection had: 9 item(s) made of steel, 12 item(s) made of ceramic, 6 item(s) made of concrete, 4 item(s) made of glass, 2 item(s) made of plastic. Then, my teacher took any item from my new collection whose color started with \\\"b\\\" or whose name had 6 characters and threw it away. Then, for any item made of plastic in my new collection, my sister changed their color to her favorite color and changed their smell to coffee. After this, my new collection had: 1 blue item(s), 2 crimson item(s), 1 gold item(s), 1 gray item(s), 1 green item(s), 2 indigo item(s), 1 khaki item(s), 1 magenta item(s), 2 maroon item(s), 1 orange item(s), 1 pink item(s), 2 purple item(s), 1 red item(s), 1 silver item(s), 2 teal item(s). Then, my uncle threw away any Mexican item in my new collection, but gave me two of the same item, one British and one Spanish (all other properties the same). Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the large items in my collection.  In my current collection, how many items have the following attributes: color is not orange, size is not large, material is not steel, smell is not popcorn, and origin is not Polish? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}