{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"51761033-6b0d-5664-8794-a7f820d74764","task_key":"bbeh~5fobject~5fproperties--full--51761033-6b0d-5664-8794-a7f820d74764","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 38 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 38 weird items that went through a few changes. Initially, I had an extra-extra-large Polish piano made of ceramic with a smell of leather, an extra-large Chinese brush made of glass with a smell of chocolate, an extra-extra-large Japanese bicycle made of plastic with a smell of vinegar, an extra-small Italian house made of ceramic with a smell of baking bread, an extra-small Chinese speaker made of glass with a smell of burning wood, an extra-extra-small Portuguese pen made of ceramic with a smell of chocolate, an extra-extra-large British car made of concrete with a smell of lavender, an extra-extra-large Portuguese umbrella made of glass with a smell of chocolate, an extra-extra-large Chinese marker made of ceramic with a smell of pine needles, a large German guitar made of glass with a smell of freshly cut grass, an extra-extra-small British box made of ceramic with a smell of citrus fruits, a small Mexican shirt made of glass with a smell of gasoline, an extra-extra-small Spanish pencil made of plastic with a smell of lavender, an extra-extra-small French pencil made of plastic with a smell of vinegar, an extra-extra-large French trash can made of concrete with a smell of burning wood, an extra-extra-small Italian table made of steel with a smell of garlic, an extra-extra-small British flower pot made of plastic with a smell of freshly cut grass, a small French knife made of steel with a smell of leather, a small Chinese wallet made of concrete with a smell of pine needles, an extra-extra-small Polish marker made of glass with a smell of citrus fruits, an extra-large Afghan banana made of ceramic with a smell of coconut, an extra-small Spanish lamp made of glass with a smell of wet dog, an extra-extra-small British canvas made of concrete with a smell of coconut, an extra-large Spanish necklace made of ceramic with a smell of baking bread, a large Afghan pencil made of steel with a smell of coconut, an extra-large German hammer made of concrete with a smell of burning wood, a small Spanish box made of ceramic with a smell of burning wood, an extra-extra-large British ball made of glass with a smell of burning wood, an extra-small German dog made of ceramic with a smell of freshly cut grass, an extra-small British house made of concrete with a smell of coffee, an extra-extra-small Spanish speaker made of glass with a smell of popcorn, a medium Chinese wallet made of concrete with a smell of freshly cut grass, an extra-extra-large Russian pen made of plastic with a smell of popcorn, a large Portuguese spoon made of plastic with a smell of baking bread, an extra-extra-large Russian dog made of ceramic with a smell of lavender, an extra-small Mexican pen made of steel with a smell of garlic, a large German guitar made of plastic with a smell of citrus fruits, a small Polish umbrella made of plastic with a smell of citrus fruits. The color of the items was respectively as follows: the first 2 were blue, the next 2 were brown, the next 2 were crimson, the next 3 were cyan, the next 2 were gold, the next 2 were gray, the next 1 was green, the next 1 was indigo, the next 3 were ivory, the next 1 was khaki, the next 1 was magenta, the next 3 were maroon, the next 2 were orange, the next 3 were red, the next 3 were silver, the next 1 was teal, the next 3 were violet, the next 2 were white, the next 1 was yellow. Then, my teacher took any item from my collection whose color started with \\\"b\\\" or whose name had 6 characters and threw it away. Then my dad threw away all item of a certain color from my new collection. After this, my new collection only had: 2 crimson item(s), 2 cyan item(s), 1 gold item(s), 1 gray item(s), 3 ivory item(s), 1 maroon item(s), 1 orange item(s), 2 red item(s), 3 silver item(s), 3 violet item(s), 1 white item(s), 1 yellow item(s). Then, for any item made of glass in my new collection, my sister changed their color to her favorite color and changed their smell to pine needles. After this, my new collection had: 1 crimson item(s), 1 cyan item(s), 1 gold item(s), 3 ivory item(s), 7 orange item(s), 1 red item(s), 2 silver item(s), 3 violet item(s), 1 white item(s), 1 yellow item(s). Then, my friend took any item with a smell of pine needles, citrus fruits or garlic in my new collection and threw it away. Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the extra-large items in my collection.  In my current collection, how many items have the following attributes: either color is ivory, or size is extra-large, or material is concrete, or smell is baking bread, or origin is Portuguese? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}