{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"447dcc45-453a-5372-8cc0-a6f3aeefa23d","task_key":"bbeh~5fobject~5fproperties--full--447dcc45-453a-5372-8cc0-a6f3aeefa23d","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 38 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 38 weird items that went through a few changes. Initially, I had an extra-extra-small Japanese drill made of ceramic with a smell of baking bread, a large Canadian lamp made of plastic with a smell of gasoline, a large British house made of plastic with a smell of garlic, a small Japanese knife made of steel with a smell of vinegar, an extra-small British bag made of ceramic with a smell of gasoline, a medium Italian fork made of steel with a smell of coconut, an extra-large German sofa made of steel with a smell of baking bread, a large Iranian trash can made of concrete with a smell of vanilla, a small Russian bicycle made of glass with a smell of wet dog, a small Mexican house made of plastic with a smell of pine needles, a large Afghan drill made of steel with a smell of citrus fruits, an extra-large Afghan lamp made of concrete with a smell of freshly cut grass, an extra-large Iranian necklace made of plastic with a smell of lavender, a medium Polish plate made of concrete with a smell of pine needles, a large Turkish spoon made of concrete with a smell of burning wood, a medium Spanish car made of steel with a smell of coffee, a small French box made of glass with a smell of freshly cut grass, a medium Turkish bird made of concrete with a smell of coconut, an extra-small Portuguese sunglasses made of plastic with a smell of gasoline, a large Portuguese bird made of plastic with a smell of garlic, an extra-extra-small French spoon made of glass with a smell of citrus fruits, a large Russian bag made of plastic with a smell of lavender, an extra-extra-small Spanish flower pot made of ceramic with a smell of garlic, a medium Iranian key made of ceramic with a smell of citrus fruits, an extra-extra-large Polish stapler made of steel with a smell of citrus fruits, an extra-small Chinese car made of glass with a smell of garlic, a small Chinese pencil made of plastic with a smell of coconut, an extra-small German guitar made of steel with a smell of coffee, an extra-small German necklace made of concrete with a smell of chocolate, a large Afghan knife made of concrete with a smell of vinegar, an extra-large Brazilian camera made of steel with a smell of vanilla, an extra-extra-small Afghan piano made of concrete with a smell of baking bread, a large Turkish speaker made of glass with a smell of chocolate, an extra-large German house made of concrete with a smell of leather, an extra-large Portuguese wallet made of steel with a smell of garlic, a small German spoon made of glass with a smell of rose, an extra-large Mexican ball made of ceramic with a smell of vanilla, an extra-small Italian sunglasses made of steel with a smell of vinegar. The color of the items was respectively as follows: the first 4 were beige, the next 2 were black, the next 2 were brown, the next 1 was crimson, the next 1 was cyan, the next 2 were gold, the next 1 was gray, the next 3 were indigo, the next 4 were ivory, the next 2 were khaki, the next 2 were magenta, the next 4 were pink, the next 2 were purple, the next 1 was teal, the next 2 were turquoise, the next 1 was violet, the next 3 were white, the next 1 was yellow. Then, for any item of color teal in my collection, my mom gave me another one of the same item, but with a khaki color, French origin, extra-extra-small size, vanilla smell, and made of concrete. Then, my friend took any item with a smell of popcorn, leather or baking bread in my new collection and threw it away. Then, my uncle threw away any Portuguese item in my new collection, but gave me two of the same item, one Brazilian and one Japanese (all other properties the same). Then, if I had an even number of an item type in my new collection (e.g. two knives, regardless of their size, color, origin, material, or smell), I threw all of them away. Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the large items in my collection.  In my current collection, how many items have the following attributes: color is not crimson, size is not large, material is not ceramic, smell is not wet dog, and origin is not Spanish? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}