{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c62f1dc4-c794-5d42-8072-15e930f43f46","task_key":"bbeh~5fobject~5fproperties--full--c62f1dc4-c794-5d42-8072-15e930f43f46","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 37 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 37 weird items that went through a few changes. Initially, I had an extra-small Spanish trash can made of ceramic with a smell of garlic, an extra-extra-small Japanese necklace made of glass with a smell of coffee, a large Russian clock made of ceramic with a smell of coconut, a medium Spanish bag made of ceramic with a smell of rose, an extra-extra-large Italian scissors made of glass with a smell of burning wood, an extra-extra-large German ball made of glass with a smell of gasoline, a small Brazilian plier made of steel with a smell of coffee, an extra-large Polish bag made of ceramic with a smell of rose, a large Russian lamp made of concrete with a smell of coffee, an extra-extra-large Afghan ball made of ceramic with a smell of rose, a medium Spanish watch made of concrete with a smell of wet dog, a large Turkish bowl made of plastic with a smell of lavender, a small Italian plate made of concrete with a smell of citrus fruits, an extra-extra-small Iranian plier made of concrete with a smell of garlic, an extra-extra-large French necklace made of glass with a smell of rose, a small British candle made of concrete with a smell of vinegar, a large French hammer made of glass with a smell of vanilla, an extra-large Chinese bird made of ceramic with a smell of vinegar, a large Brazilian car made of glass with a smell of lavender, an extra-extra-large Russian box made of steel with a smell of popcorn, an extra-extra-large Afghan bird made of concrete with a smell of citrus fruits, a small Brazilian flower pot made of concrete with a smell of popcorn, a medium Chinese drill made of glass with a smell of leather, a large Mexican key made of concrete with a smell of pine needles, an extra-extra-large Canadian bird made of plastic with a smell of burning wood, an extra-large Italian fork made of plastic with a smell of pine needles, a medium French clock made of concrete with a smell of vanilla, an extra-extra-large Canadian plate made of concrete with a smell of citrus fruits, an extra-small Iranian plate made of concrete with a smell of lavender, a small Turkish screwdriver made of steel with a smell of rose, an extra-extra-large German house made of concrete with a smell of garlic, an extra-large Mexican earring made of concrete with a smell of pine needles, a large Iranian trash can made of steel with a smell of burning wood, a medium French watch made of ceramic with a smell of vinegar, an extra-small Japanese vase made of steel with a smell of rose, an extra-small Mexican pen made of concrete with a smell of freshly cut grass, an extra-small French marker made of steel with a smell of leather. The color of the items was respectively as follows: the first 2 were blue, the next 1 was crimson, the next 2 were gold, the next 2 were gray, the next 4 were green, the next 2 were indigo, the next 4 were ivory, the next 1 was khaki, the next 1 was magenta, the next 5 were maroon, the next 2 were pink, the next 1 was purple, the next 2 were red, the next 1 was silver, the next 4 were teal, the next 1 was turquoise, the next 1 was violet, the next 1 was yellow. Then, my uncle threw away any Mexican item in my collection, but gave me two of the same item, one Brazilian and one Canadian (all other properties the same). Then, my teacher took any item from my new collection whose color started with \\\"b\\\" or whose name had 6 characters and threw it away. Then, if I had an even number of an item type in my new collection (e.g. two knives, regardless of their size, color, origin, material, or smell), I threw all of them away. Then, for any item with a smell of coconut, garlic, or baking bread in my new collection, my cousin gifted me another one of that item with the same properties but with her favorite smell (different from the three smells above). After this, my new collection had: 3 item(s) with burning wood smell, 1 item(s) with coffee smell, 3 item(s) with lavender smell, 3 item(s) with citrus fruits smell, 3 item(s) with rose smell, 2 item(s) with vinegar smell, 2 item(s) with popcorn smell, 1 item(s) with leather smell, 1 item(s) with pine needles smell, 1 item(s) with garlic smell. Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the small items in my collection.  In my current collection, how many items have the following attributes: either color is turquoise, or size is small, or material is concrete, or smell is burning wood, or origin is Italian? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}