{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"23be5817-3fbc-5264-9a08-ce2505358721","task_key":"bbeh~5fobject~5fproperties--full--23be5817-3fbc-5264-9a08-ce2505358721","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 37 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 37 weird items that went through a few changes. Initially, I had a medium Iranian ball made of glass with a smell of wet dog, an extra-extra-large French ruler made of concrete with a smell of burning wood, a small Italian flower pot made of ceramic with a smell of popcorn, an extra-extra-large Russian banana made of steel with a smell of freshly cut grass, a small Canadian wallet made of plastic with a smell of lavender, a large Russian sunglasses made of concrete with a smell of pine needles, an extra-large Canadian pencil made of concrete with a smell of wet dog, a large French apple made of ceramic with a smell of chocolate, an extra-extra-large Russian house made of plastic with a smell of vinegar, an extra-extra-small Italian plier made of plastic with a smell of burning wood, a small Mexican dog made of plastic with a smell of chocolate, an extra-extra-small British candle made of glass with a smell of coffee, an extra-extra-small Iranian marker made of glass with a smell of gasoline, an extra-small Japanese marker made of concrete with a smell of lavender, an extra-extra-large Portuguese calculator made of plastic with a smell of leather, a small Canadian piano made of plastic with a smell of coconut, a large Mexican box made of glass with a smell of pine needles, an extra-extra-large Mexican wallet made of plastic with a smell of gasoline, an extra-large German dog made of steel with a smell of coconut, an extra-large Spanish bag made of steel with a smell of coconut, an extra-large Spanish car made of glass with a smell of garlic, a large French camera made of concrete with a smell of rose, an extra-extra-large French guitar made of steel with a smell of rose, a medium British ruler made of concrete with a smell of garlic, a large Canadian saw made of steel with a smell of coffee, an extra-large Portuguese banana made of plastic with a smell of freshly cut grass, an extra-extra-small Italian bed made of steel with a smell of citrus fruits, a small Spanish bed made of concrete with a smell of gasoline, an extra-extra-large Mexican necklace made of plastic with a smell of burning wood, a medium Italian bicycle made of steel with a smell of gasoline, a medium Russian umbrella made of plastic with a smell of vanilla, an extra-small French bed made of steel with a smell of freshly cut grass, an extra-extra-large French candle made of concrete with a smell of burning wood, a medium Brazilian wallet made of steel with a smell of rose, an extra-extra-small Spanish shirt made of steel with a smell of vanilla, a medium Polish scissors made of ceramic with a smell of coconut, a large Afghan camera made of glass with a smell of coconut. The color of the items was respectively as follows: the first 1 was blue, the next 1 was crimson, the next 3 were cyan, the next 1 was green, the next 1 was indigo, the next 2 were ivory, the next 3 were khaki, the next 4 were orange, the next 4 were pink, the next 3 were purple, the next 3 were red, the next 4 were silver, the next 3 were turquoise, the next 1 was violet, the next 2 were white, the next 1 was yellow. Then my dad threw away all item of a certain color from my collection. After this, my collection only had: 1 blue item(s), 1 crimson item(s), 3 cyan item(s), 1 green item(s), 1 indigo item(s), 2 ivory item(s), 3 khaki item(s), 4 orange item(s), 3 purple item(s), 3 red item(s), 4 silver item(s), 3 turquoise item(s), 1 violet item(s), 2 white item(s), 1 yellow item(s). Then, if I had an even number of an item type in my new collection (e.g. two knives, regardless of their size, color, origin, material, or smell), I threw all of them away. Then, my uncle threw away any Portuguese item in my new collection, but gave me two of the same item, one Canadian and one Polish (all other properties the same). Then, for any item of color ivory in my new collection, my mom gave me another one of the same item, but with a green color, Canadian origin, extra-extra-small size, coffee smell, and made of steel. Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the medium items in my collection.  In my current collection, how many items have the following attributes: color is not purple, size is not medium, material is not ceramic, smell is not vinegar, and origin is not Iranian? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}