{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0189ba89-c9a8-5984-b250-0944df49724b","task_key":"bbeh~5fobject~5fproperties--full--0189ba89-c9a8-5984-b250-0944df49724b","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 32 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 32 weird items that went through a few changes. Initially, I had an extra-extra-small Turkish screwdriver made of concrete with a smell of freshly cut grass, an extra-extra-large Turkish plier made of ceramic with a smell of pine needles, a medium Japanese car made of plastic with a smell of freshly cut grass, an extra-small German house made of glass with a smell of pine needles, a large Mexican car made of ceramic with a smell of pine needles, a small Afghan dog made of steel with a smell of coconut, a large Japanese watch made of concrete with a smell of wet dog, an extra-extra-large Mexican knife made of plastic with a smell of popcorn, an extra-extra-large Canadian spoon made of glass with a smell of popcorn, an extra-large British marker made of ceramic with a smell of gasoline, a medium Iranian bag made of plastic with a smell of popcorn, a large German scissors made of plastic with a smell of vinegar, a large Turkish hammer made of plastic with a smell of garlic, an extra-large Spanish box made of concrete with a smell of coconut, an extra-extra-large Afghan clock made of plastic with a smell of chocolate, a small Russian bird made of glass with a smell of popcorn, a small Polish banana made of steel with a smell of leather, an extra-large Brazilian key made of steel with a smell of coconut, a medium Russian apple made of ceramic with a smell of baking bread, a small Afghan saw made of plastic with a smell of vanilla, an extra-large German knife made of ceramic with a smell of coffee, an extra-extra-large German bowl made of steel with a smell of freshly cut grass, a medium Afghan watch made of plastic with a smell of garlic, an extra-large Iranian drill made of steel with a smell of burning wood, a medium Italian spoon made of steel with a smell of vinegar, a medium Polish bag made of concrete with a smell of garlic, an extra-small Mexican house made of concrete with a smell of freshly cut grass, an extra-large Italian shoe made of concrete with a smell of freshly cut grass, an extra-small Mexican candle made of glass with a smell of burning wood, an extra-large Spanish flower pot made of plastic with a smell of popcorn, a large Italian banana made of steel with a smell of lavender, an extra-small Japanese calculator made of steel with a smell of wet dog. The color of the items was respectively as follows: the first 5 were beige, the next 1 was black, the next 1 was blue, the next 1 was cyan, the next 1 was gray, the next 2 were indigo, the next 3 were ivory, the next 1 was maroon, the next 1 was orange, the next 3 were purple, the next 3 were red, the next 3 were silver, the next 1 was teal, the next 1 was turquoise, the next 2 were violet, the next 2 were white, the next 1 was yellow. Then my dad threw away all item of a certain color from my collection. After this, my collection only had: 5 beige item(s), 1 black item(s), 1 blue item(s), 1 cyan item(s), 1 gray item(s), 2 indigo item(s), 3 ivory item(s), 1 maroon item(s), 1 orange item(s), 3 purple item(s), 3 red item(s), 1 teal item(s), 1 turquoise item(s), 2 violet item(s), 2 white item(s), 1 yellow item(s). Then, for any item made of concrete in my new collection, my sister changed their color to her favorite color and changed their smell to garlic. After this, my new collection had: 4 beige item(s), 1 black item(s), 6 blue item(s), 1 cyan item(s), 1 gray item(s), 2 indigo item(s), 2 ivory item(s), 1 maroon item(s), 1 orange item(s), 3 purple item(s), 3 red item(s), 1 violet item(s), 2 white item(s), 1 yellow item(s). Then, for any item of color orange in my new collection, my mom gave me another one of the same item, but with a blue color, Turkish origin, medium size, vanilla smell, and made of ceramic. Then, my teacher took any item from my new collection whose color started with \\\"b\\\" or whose name had 6 characters and threw it away. Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the large items in my collection.  In my current collection, how many items have the following attributes: either color is gray, or size is large, or material is glass, or smell is coffee, or origin is Mexican? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}