{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0c9bdb74-51ec-517f-a954-0fc884f5f4f7","task_key":"bbeh~5fobject~5fproperties--full--0c9bdb74-51ec-517f-a954-0fc884f5f4f7","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 37 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 37 weird items that went through a few changes. Initially, I had a small British shoe made of ceramic with a smell of freshly cut grass, a small Canadian spoon made of glass with a smell of rose, a small Japanese shoe made of ceramic with a smell of burning wood, a large Brazilian canvas made of concrete with a smell of lavender, an extra-extra-large Mexican sunglasses made of steel with a smell of citrus fruits, a medium British pencil made of concrete with a smell of lavender, a medium Brazilian piano made of ceramic with a smell of wet dog, an extra-extra-large Afghan ring made of glass with a smell of freshly cut grass, an extra-large British calculator made of concrete with a smell of rose, an extra-large Italian car made of glass with a smell of burning wood, a large Portuguese brush made of plastic with a smell of popcorn, an extra-small Afghan guitar made of glass with a smell of garlic, an extra-large Russian stapler made of steel with a smell of baking bread, an extra-extra-small Japanese lamp made of plastic with a smell of rose, an extra-extra-large German vase made of glass with a smell of leather, a large Russian drill made of plastic with a smell of pine needles, an extra-small German shoe made of steel with a smell of pine needles, an extra-extra-large Italian knife made of ceramic with a smell of burning wood, an extra-extra-small French plier made of plastic with a smell of citrus fruits, a large Turkish key made of glass with a smell of coconut, an extra-extra-small Afghan scissors made of plastic with a smell of pine needles, a small British brush made of glass with a smell of lavender, a medium German box made of steel with a smell of baking bread, an extra-small German book made of glass with a smell of baking bread, an extra-small German bag made of concrete with a smell of burning wood, an extra-large German car made of steel with a smell of citrus fruits, an extra-extra-large Spanish umbrella made of steel with a smell of vinegar, an extra-large French calculator made of glass with a smell of chocolate, a small Iranian bag made of ceramic with a smell of vanilla, a small Afghan hammer made of concrete with a smell of vinegar, an extra-extra-small Japanese candle made of glass with a smell of leather, a small Spanish bicycle made of glass with a smell of baking bread, an extra-large Mexican guitar made of concrete with a smell of gasoline, a medium Turkish car made of glass with a smell of chocolate, a medium French bottle made of glass with a smell of coconut, an extra-large Japanese scissors made of ceramic with a smell of gasoline, an extra-large Turkish apple made of ceramic with a smell of lavender. The color of the items was respectively as follows: the first 1 was beige, the next 1 was black, the next 1 was blue, the next 2 were brown, the next 5 were crimson, the next 1 was gold, the next 2 were green, the next 1 was indigo, the next 4 were khaki, the next 2 were magenta, the next 3 were maroon, the next 1 was orange, the next 4 were purple, the next 2 were red, the next 1 was silver, the next 1 was turquoise, the next 1 was violet, the next 2 were white, the next 2 were yellow. Then, if I had an even number of an item type in my collection (e.g. two knives, regardless of their size, color, origin, material, or smell), I threw all of them away. Then, my friend took any item with a smell of wet dog, vinegar or citrus fruits in my new collection and threw it away. Then, for each item of size small in my new collection, my aunt took it but then gave me another one of the same item made of her favorite material and with a smell of freshly cut grass (all other properties the same). After this, my new collection had: 12 item(s) made of glass, 2 item(s) made of concrete, 3 item(s) made of steel, 2 item(s) made of plastic, 2 item(s) made of ceramic. Then, my uncle threw away any Brazilian item in my new collection, but gave me two of the same item, one Russian and one Japanese (all other properties the same). Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the extra-extra-small items in my collection.  In my current collection, how many items have the following attributes: either color is indigo, or size is extra-extra-small, or material is plastic, or smell is baking bread, or origin is Turkish? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}