{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"31aa402d-5993-5423-81ba-2cc6ec6169be","task_key":"bbeh~5fobject~5fproperties--full--31aa402d-5993-5423-81ba-2cc6ec6169be","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 46 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 46 weird items that went through a few changes. Initially, I had a large Iranian piano made of steel with a smell of rose, an extra-extra-small Mexican ring made of glass with a smell of coconut, a small Brazilian plate made of plastic with a smell of wet dog, a medium Polish car made of concrete with a smell of popcorn, an extra-large British bed made of plastic with a smell of vinegar, a small Russian bottle made of glass with a smell of citrus fruits, an extra-large Afghan screwdriver made of concrete with a smell of chocolate, a medium Chinese speaker made of plastic with a smell of leather, an extra-extra-small Mexican brush made of steel with a smell of leather, a medium Russian car made of steel with a smell of rose, an extra-small French ball made of steel with a smell of lavender, a medium Turkish saw made of concrete with a smell of popcorn, an extra-large Japanese camera made of ceramic with a smell of rose, an extra-extra-small Afghan scissors made of plastic with a smell of popcorn, a medium Mexican plate made of steel with a smell of gasoline, an extra-large Russian fork made of ceramic with a smell of rose, a small Afghan scissors made of plastic with a smell of garlic, a medium German hammer made of glass with a smell of coconut, a large Mexican bottle made of ceramic with a smell of coconut, a small Portuguese clock made of plastic with a smell of rose, a large Japanese plate made of steel with a smell of lavender, a large Iranian jar made of plastic with a smell of rose, an extra-small German wrench made of ceramic with a smell of gasoline, a large Mexican vase made of plastic with a smell of freshly cut grass, an extra-extra-large Japanese banana made of glass with a smell of vinegar, an extra-large Polish watch made of steel with a smell of coffee, a small Mexican watch made of ceramic with a smell of lavender, a medium Spanish car made of plastic with a smell of popcorn, a small Brazilian shoe made of ceramic with a smell of citrus fruits, a small Spanish wallet made of glass with a smell of freshly cut grass, an extra-small Italian table made of glass with a smell of garlic, a large German candle made of ceramic with a smell of citrus fruits, an extra-small British box made of plastic with a smell of gasoline, an extra-small Canadian bottle made of concrete with a smell of popcorn, a small Brazilian trash can made of ceramic with a smell of popcorn, a small Turkish plier made of plastic with a smell of vanilla, a medium Chinese ball made of concrete with a smell of rose, a medium Iranian box made of concrete with a smell of citrus fruits, an extra-large French brush made of steel with a smell of burning wood, an extra-extra-large British drill made of concrete with a smell of vanilla, an extra-small Afghan stapler made of plastic with a smell of popcorn, an extra-extra-large Brazilian drill made of ceramic with a smell of vanilla, an extra-extra-small Chinese dog made of steel with a smell of freshly cut grass, an extra-extra-large Japanese bottle made of plastic with a smell of lavender, an extra-small Italian candle made of concrete with a smell of lavender, an extra-extra-small Chinese table made of concrete with a smell of citrus fruits. The color of the items was respectively as follows: the first 1 was black, the next 2 were blue, the next 3 were brown, the next 1 was crimson, the next 1 was cyan, the next 1 was gold, the next 3 were gray, the next 5 were green, the next 4 were indigo, the next 1 was ivory, the next 1 was khaki, the next 1 was magenta, the next 2 were maroon, the next 2 were orange, the next 1 was pink, the next 6 were red, the next 1 was teal, the next 1 was turquoise, the next 4 were violet, the next 2 were white, the next 3 were yellow. Then, my friend took any item with a smell of gasoline, burning wood or chocolate in my collection and threw it away. Then, my teacher took any item from my new collection whose color started with \\\"b\\\" or whose name had 6 characters and threw it away. Then, if I had an even number of an item type in my new collection (e.g. two knives, regardless of their size, color, origin, material, or smell), I threw all of them away. Then, for any item with a smell of burning wood, freshly cut grass, or gasoline in my new collection, my cousin gifted me another one of that item with the same properties but with her favorite smell (different from the three smells above). After this, my new collection had: 2 item(s) with leather smell, 5 item(s) with popcorn smell, 3 item(s) with rose smell, 1 item(s) with lavender smell, 2 item(s) with freshly cut grass smell, 2 item(s) with citrus fruits smell, 1 item(s) with vanilla smell. Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the small items in my collection.  In my current collection, how many items have the following attributes: color is not violet, size is not small, material is not concrete, smell is not vanilla, and origin is not Portuguese? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}