{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e74223b0-c019-5563-8e81-7bdf07029389","task_key":"bbeh~5fobject~5fproperties--full--e74223b0-c019-5563-8e81-7bdf07029389","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 30 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 30 weird items that went through a few changes. Initially, I had an extra-large Iranian piano made of plastic with a smell of garlic, a small French knife made of steel with a smell of lavender, an extra-extra-small Portuguese book made of steel with a smell of coconut, an extra-extra-large Italian candle made of ceramic with a smell of chocolate, an extra-extra-large Brazilian hat made of glass with a smell of leather, an extra-extra-large Iranian box made of steel with a smell of gasoline, an extra-large Chinese marker made of steel with a smell of vanilla, a medium British wallet made of plastic with a smell of pine needles, an extra-extra-small Brazilian piano made of steel with a smell of freshly cut grass, a medium Iranian bottle made of plastic with a smell of chocolate, a medium Afghan box made of plastic with a smell of citrus fruits, an extra-small Chinese bottle made of glass with a smell of lavender, an extra-small Chinese cup made of plastic with a smell of chocolate, an extra-extra-large Polish umbrella made of glass with a smell of citrus fruits, an extra-extra-large Chinese ball made of steel with a smell of freshly cut grass, a large Spanish table made of ceramic with a smell of garlic, an extra-extra-large Canadian chair made of steel with a smell of pine needles, an extra-extra-large Afghan lamp made of plastic with a smell of pine needles, an extra-large German hat made of concrete with a smell of pine needles, a medium British pencil made of steel with a smell of citrus fruits, a large Brazilian sunglasses made of steel with a smell of baking bread, a medium Brazilian bicycle made of plastic with a smell of gasoline, an extra-large British bottle made of plastic with a smell of baking bread, a large Chinese calculator made of ceramic with a smell of citrus fruits, an extra-small Iranian candle made of concrete with a smell of garlic, an extra-small Italian cup made of glass with a smell of rose, an extra-large Chinese hammer made of glass with a smell of baking bread, an extra-small French apple made of steel with a smell of garlic, an extra-extra-large German bicycle made of concrete with a smell of coffee, a small Italian plate made of plastic with a smell of rose. The color of the items was respectively as follows: the first 3 were black, the next 2 were brown, the next 2 were crimson, the next 1 was gold, the next 2 were gray, the next 2 were green, the next 3 were ivory, the next 2 were khaki, the next 1 was magenta, the next 1 was orange, the next 1 was purple, the next 4 were red, the next 1 was turquoise, the next 1 was violet, the next 2 were white, the next 2 were yellow. Then, if I had an even number of an item type in my collection (e.g. two knives, regardless of their size, color, origin, material, or smell), I threw all of them away. Then, for any item of color red in my new collection, my mom gave me another one of the same item, but with a magenta color, Chinese origin, medium size, pine needles smell, and made of steel. Then, for any item made of steel in my new collection, my sister changed their color to her favorite color and changed their smell to popcorn. After this, my new collection had: 11 brown item(s), 1 gold item(s), 1 gray item(s), 1 green item(s), 1 ivory item(s), 1 khaki item(s), 1 magenta item(s), 2 red item(s), 1 white item(s), 1 yellow item(s). Then, for each item of size extra-small in my new collection, my aunt took it but then gave me another one of the same item made of her favorite material and with a smell of lavender (all other properties the same). After this, my new collection had: 10 item(s) made of steel, 7 item(s) made of plastic, 2 item(s) made of glass, 2 item(s) made of ceramic. Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the extra-large items in my collection.  In my current collection, how many items have the following attributes: color is not magenta, size is not extra-large, material is not steel, smell is not citrus fruits, and origin is not Afghan? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}