{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1b9036e8-5c4a-5eec-be42-9f0c0bec70e9","task_key":"bbeh~5fobject~5fproperties--full--1b9036e8-5c4a-5eec-be42-9f0c0bec70e9","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 34 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 34 weird items that went through a few changes. Initially, I had a small French fork made of steel with a smell of rose, an extra-extra-large Portuguese marker made of concrete with a smell of garlic, an extra-large Chinese hammer made of glass with a smell of pine needles, an extra-extra-large British pen made of concrete with a smell of citrus fruits, a small Italian vase made of ceramic with a smell of vinegar, an extra-extra-large Afghan apple made of glass with a smell of freshly cut grass, an extra-large German pencil made of ceramic with a smell of wet dog, a small Iranian ruler made of concrete with a smell of burning wood, an extra-large Russian sunglasses made of steel with a smell of vanilla, a large Spanish plier made of glass with a smell of chocolate, a large Russian plier made of steel with a smell of popcorn, an extra-extra-large Italian vase made of steel with a smell of vanilla, an extra-extra-large Brazilian bottle made of glass with a smell of freshly cut grass, an extra-extra-large French dog made of glass with a smell of coconut, a medium Polish box made of plastic with a smell of lavender, an extra-extra-large Brazilian vase made of concrete with a smell of chocolate, a medium British fork made of ceramic with a smell of vanilla, a small Mexican plate made of ceramic with a smell of rose, an extra-extra-large Brazilian screwdriver made of concrete with a smell of baking bread, an extra-extra-large Afghan speaker made of concrete with a smell of coffee, an extra-small Spanish plier made of ceramic with a smell of freshly cut grass, an extra-extra-large German piano made of glass with a smell of chocolate, an extra-small Afghan camera made of plastic with a smell of coffee, a large Turkish fork made of glass with a smell of rose, an extra-small Spanish house made of glass with a smell of citrus fruits, a large Russian plate made of plastic with a smell of vanilla, an extra-large British bag made of steel with a smell of burning wood, an extra-extra-small Portuguese chair made of plastic with a smell of lavender, an extra-extra-large Russian book made of concrete with a smell of vanilla, a medium Afghan apple made of glass with a smell of chocolate, a large Portuguese clock made of concrete with a smell of citrus fruits, an extra-extra-large Canadian piano made of concrete with a smell of garlic, an extra-extra-small Mexican clock made of glass with a smell of leather, a medium Mexican plate made of ceramic with a smell of pine needles. The color of the items was respectively as follows: the first 2 were beige, the next 2 were black, the next 2 were blue, the next 2 were crimson, the next 1 was cyan, the next 1 was gray, the next 3 were indigo, the next 2 were ivory, the next 1 was magenta, the next 1 was maroon, the next 2 were orange, the next 3 were pink, the next 2 were purple, the next 2 were red, the next 1 was silver, the next 1 was teal, the next 1 was turquoise, the next 3 were violet, the next 2 were white. Then, my uncle threw away any Spanish item in my collection, but gave me two of the same item, one Portuguese and one Italian (all other properties the same). Then, my brother replaced any item of size small in my new collection with an exact copy but with one specific size (different from small) and another exact copy but with color teal. After this, my new collection had: 6 extra-extra-small item(s), 12 extra-extra-large item(s), 4 extra-large item(s), 6 large item(s), 4 medium item(s), 5 extra-small item(s), 4 small item(s). Then, if I had an even number of an item type in my new collection (e.g. two knives, regardless of their size, color, origin, material, or smell), I threw all of them away. Then my dad threw away all item of a certain color from my new collection. After this, my new collection only had: 1 beige item(s), 2 black item(s), 1 crimson item(s), 1 cyan item(s), 2 gray item(s), 2 indigo item(s), 2 ivory item(s), 1 orange item(s), 3 pink item(s), 1 silver item(s), 1 teal item(s), 1 turquoise item(s). Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the extra-extra-large items in my collection.  In my current collection, how many items have the following attributes: either color is orange, or size is extra-extra-large, or material is ceramic, or smell is coffee, or origin is Russian? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}