{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9a1ef409-641b-5a41-a6cf-856075864fc1","task_key":"bbeh~5fobject~5fproperties--full--9a1ef409-641b-5a41-a6cf-856075864fc1","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 33 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 33 weird items that went through a few changes. Initially, I had a large Portuguese hammer made of glass with a smell of citrus fruits, a medium British cup made of plastic with a smell of gasoline, a medium Turkish box made of concrete with a smell of baking bread, an extra-extra-large Polish drill made of plastic with a smell of lavender, an extra-extra-large Chinese pencil made of ceramic with a smell of baking bread, an extra-extra-large Iranian stapler made of steel with a smell of pine needles, an extra-extra-large Chinese table made of glass with a smell of vinegar, a large British ball made of steel with a smell of coffee, a large Japanese vase made of ceramic with a smell of citrus fruits, an extra-large Japanese stapler made of glass with a smell of burning wood, a medium Afghan bicycle made of plastic with a smell of leather, an extra-small Iranian saw made of plastic with a smell of gasoline, an extra-small Brazilian marker made of plastic with a smell of popcorn, a large Turkish cup made of ceramic with a smell of gasoline, a small German lamp made of plastic with a smell of lavender, an extra-extra-small Afghan bowl made of glass with a smell of popcorn, an extra-large Portuguese knife made of concrete with a smell of chocolate, a small Spanish guitar made of ceramic with a smell of popcorn, a medium Iranian necklace made of glass with a smell of vinegar, a small French screwdriver made of concrete with a smell of lavender, an extra-large Spanish calculator made of ceramic with a smell of lavender, a large Afghan piano made of steel with a smell of popcorn, a medium British key made of glass with a smell of chocolate, an extra-small Turkish guitar made of glass with a smell of rose, an extra-extra-small Brazilian scissors made of glass with a smell of pine needles, a medium Portuguese vase made of plastic with a smell of gasoline, a medium Portuguese plate made of plastic with a smell of vinegar, a large Canadian shirt made of ceramic with a smell of popcorn, an extra-extra-small Brazilian shirt made of ceramic with a smell of pine needles, a large Chinese banana made of steel with a smell of vanilla, an extra-extra-small Canadian bed made of ceramic with a smell of leather, an extra-large Brazilian book made of steel with a smell of vanilla, an extra-large Japanese umbrella made of plastic with a smell of baking bread. The color of the items was respectively as follows: the first 5 were beige, the next 1 was black, the next 3 were blue, the next 1 was brown, the next 2 were cyan, the next 1 was gray, the next 1 was green, the next 3 were indigo, the next 3 were khaki, the next 1 was magenta, the next 1 was orange, the next 1 was pink, the next 1 was purple, the next 1 was red, the next 3 were silver, the next 2 were turquoise, the next 2 were violet, the next 1 was yellow. Then my dad threw away all item of a certain color from my collection. After this, my collection only had: 5 beige item(s), 1 black item(s), 3 blue item(s), 1 brown item(s), 2 cyan item(s), 1 gray item(s), 1 green item(s), 3 indigo item(s), 3 khaki item(s), 1 magenta item(s), 1 orange item(s), 1 pink item(s), 1 purple item(s), 3 silver item(s), 2 turquoise item(s), 2 violet item(s), 1 yellow item(s). Then, my uncle threw away any Japanese item in my new collection, but gave me two of the same item, one Chinese and one Brazilian (all other properties the same). Then, for any item with a smell of vanilla, gasoline, or wet dog in my new collection, my cousin gifted me another one of that item with the same properties but with her favorite smell (different from the three smells above). After this, my new collection had: 3 item(s) with citrus fruits smell, 4 item(s) with gasoline smell, 10 item(s) with baking bread smell, 4 item(s) with lavender smell, 2 item(s) with pine needles smell, 3 item(s) with vinegar smell, 1 item(s) with coffee smell, 2 item(s) with burning wood smell, 2 item(s) with leather smell, 5 item(s) with popcorn smell, 2 item(s) with chocolate smell, 1 item(s) with rose smell, 2 item(s) with vanilla smell. Then, for any item of color beige in my new collection, my mom gave me another one of the same item, but with a khaki color, German origin, large size, burning wood smell, and made of concrete. Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the small items in my collection.  In my current collection, how many items have the following attributes: color is not violet, size is not small, material is not steel, smell is not citrus fruits, and origin is not Afghan? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}