{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"db716e9c-bfe4-5b17-988a-639d0ae33150","task_key":"bbeh~5fobject~5fproperties--full--db716e9c-bfe4-5b17-988a-639d0ae33150","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 32 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 32 weird items that went through a few changes. Initially, I had a large Mexican drill made of glass with a smell of vinegar, a medium Mexican watch made of plastic with a smell of coconut, a medium Portuguese car made of plastic with a smell of gasoline, a large Japanese drill made of steel with a smell of coconut, an extra-large Afghan plier made of concrete with a smell of pine needles, an extra-extra-large German guitar made of steel with a smell of chocolate, a large Portuguese calculator made of steel with a smell of vanilla, a large Mexican pen made of glass with a smell of wet dog, an extra-extra-small Iranian bed made of plastic with a smell of pine needles, an extra-large Polish speaker made of ceramic with a smell of wet dog, an extra-small German calculator made of concrete with a smell of lavender, an extra-extra-small Canadian key made of glass with a smell of baking bread, a large Iranian ring made of ceramic with a smell of wet dog, a large British trash can made of plastic with a smell of freshly cut grass, an extra-large Afghan guitar made of plastic with a smell of freshly cut grass, an extra-extra-large Mexican apple made of ceramic with a smell of baking bread, a small Portuguese key made of steel with a smell of burning wood, a medium Spanish car made of glass with a smell of pine needles, an extra-extra-small Chinese box made of concrete with a smell of coffee, a large German sofa made of plastic with a smell of rose, an extra-large British bird made of concrete with a smell of lavender, a small Chinese fork made of plastic with a smell of chocolate, an extra-small Spanish knife made of steel with a smell of vanilla, an extra-extra-large Japanese stapler made of glass with a smell of vanilla, a large Italian table made of glass with a smell of rose, a large Canadian lamp made of glass with a smell of chocolate, an extra-small Italian plate made of concrete with a smell of wet dog, a large French flower pot made of steel with a smell of popcorn, an extra-extra-small Afghan hat made of glass with a smell of garlic, an extra-large Afghan brush made of concrete with a smell of coconut, an extra-small Russian bowl made of ceramic with a smell of leather, an extra-extra-small Brazilian drill made of glass with a smell of burning wood. The color of the items was respectively as follows: the first 1 was beige, the next 3 were black, the next 6 were blue, the next 1 was brown, the next 1 was crimson, the next 2 were gold, the next 3 were green, the next 1 was khaki, the next 1 was magenta, the next 2 were maroon, the next 4 were orange, the next 1 was pink, the next 1 was purple, the next 2 were red, the next 2 were silver, the next 1 was turquoise. Then, for any item of color orange in my collection, my mom gave me another one of the same item, but with a magenta color, German origin, extra-extra-small size, pine needles smell, and made of plastic. Then my dad threw away all item of a certain color from my new collection. After this, my new collection only had: 1 beige item(s), 6 blue item(s), 1 brown item(s), 1 crimson item(s), 2 gold item(s), 3 green item(s), 1 khaki item(s), 5 magenta item(s), 2 maroon item(s), 4 orange item(s), 1 pink item(s), 1 purple item(s), 2 red item(s), 2 silver item(s), 1 turquoise item(s). Then, my friend took any item with a smell of vanilla, rose or vinegar in my new collection and threw it away. Then, for each item of size small in my new collection, my aunt took it but then gave me another one of the same item made of her favorite material and with a smell of garlic (all other properties the same). After this, my new collection had: 8 item(s) made of concrete, 2 item(s) made of steel, 6 item(s) made of glass, 7 item(s) made of plastic, 4 item(s) made of ceramic. Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the extra-large items in my collection.  In my current collection, how many items have the following attributes: color is not gold, size is not extra-large, material is not steel, smell is not coffee, and origin is not Italian? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}