{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1795f25c-50eb-53c5-b894-b45bb447c137","task_key":"bbeh~5fobject~5fproperties--full--1795f25c-50eb-53c5-b894-b45bb447c137","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 30 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 30 weird items that went through a few changes. Initially, I had an extra-extra-large Turkish scissors made of glass with a smell of gasoline, a small Spanish sunglasses made of ceramic with a smell of burning wood, an extra-large Italian piano made of concrete with a smell of gasoline, an extra-extra-small German jar made of steel with a smell of garlic, an extra-large Afghan spoon made of concrete with a smell of rose, an extra-large German car made of ceramic with a smell of rose, a small French brush made of steel with a smell of popcorn, an extra-extra-small German flower pot made of concrete with a smell of lavender, an extra-extra-small Afghan speaker made of ceramic with a smell of popcorn, a large Chinese chair made of glass with a smell of freshly cut grass, an extra-small Mexican hammer made of glass with a smell of lavender, an extra-small Turkish drill made of concrete with a smell of baking bread, an extra-large French candle made of steel with a smell of citrus fruits, an extra-large Iranian banana made of concrete with a smell of burning wood, an extra-large German wallet made of concrete with a smell of coffee, a medium Turkish camera made of concrete with a smell of pine needles, an extra-extra-large Portuguese piano made of glass with a smell of pine needles, an extra-small Polish camera made of concrete with a smell of popcorn, a large Spanish chair made of ceramic with a smell of leather, an extra-large French clock made of plastic with a smell of lavender, a large Chinese shirt made of ceramic with a smell of gasoline, a small Brazilian shirt made of concrete with a smell of burning wood, a medium French pencil made of concrete with a smell of burning wood, a small Afghan banana made of ceramic with a smell of pine needles, an extra-extra-large French sunglasses made of concrete with a smell of pine needles, a small Afghan canvas made of plastic with a smell of rose, an extra-large British chair made of glass with a smell of burning wood, an extra-small Spanish drill made of concrete with a smell of burning wood, an extra-small French hammer made of steel with a smell of gasoline, a small Polish screwdriver made of ceramic with a smell of popcorn. The color of the items was respectively as follows: the first 2 were beige, the next 2 were black, the next 1 was brown, the next 1 was crimson, the next 2 were cyan, the next 4 were gold, the next 1 was gray, the next 1 was green, the next 1 was indigo, the next 1 was ivory, the next 1 was khaki, the next 1 was magenta, the next 3 were pink, the next 1 was purple, the next 2 were red, the next 2 were teal, the next 2 were turquoise, the next 1 was white, the next 1 was yellow. Then, if I had an even number of an item type in my collection (e.g. two knives, regardless of their size, color, origin, material, or smell), I threw all of them away. Then my dad threw away all item of a certain color from my new collection. After this, my new collection only had: 1 beige item(s), 1 black item(s), 1 brown item(s), 1 crimson item(s), 2 cyan item(s), 2 gold item(s), 1 gray item(s), 1 indigo item(s), 2 pink item(s), 1 red item(s), 1 turquoise item(s), 1 yellow item(s). Then, for any item made of concrete in my new collection, my sister changed their color to her favorite color and changed their smell to citrus fruits. After this, my new collection had: 1 beige item(s), 1 black item(s), 4 blue item(s), 1 crimson item(s), 1 cyan item(s), 2 gold item(s), 1 gray item(s), 2 pink item(s), 1 turquoise item(s), 1 yellow item(s). Then, my uncle threw away any Turkish item in my new collection, but gave me two of the same item, one Spanish and one Chinese (all other properties the same). Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the small items in my collection.  In my current collection, how many items have the following attributes: color is not gold, size is not small, material is not concrete, smell is not citrus fruits, and origin is not German? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}