{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ef23814d-6ffb-57c3-8b7f-c22334c87d5e","task_key":"bbeh~5fobject~5fproperties--full--ef23814d-6ffb-57c3-8b7f-c22334c87d5e","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 33 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 33 weird items that went through a few changes. Initially, I had an extra-small Polish candle made of plastic with a smell of citrus fruits, a large Afghan hat made of concrete with a smell of leather, an extra-extra-large British guitar made of concrete with a smell of baking bread, an extra-extra-small Russian hammer made of plastic with a smell of freshly cut grass, a small British banana made of ceramic with a smell of vinegar, a medium Spanish ruler made of glass with a smell of citrus fruits, a medium German cup made of concrete with a smell of coffee, an extra-extra-small Turkish plier made of plastic with a smell of pine needles, a medium French ball made of plastic with a smell of freshly cut grass, an extra-small Iranian guitar made of concrete with a smell of pine needles, an extra-extra-large Brazilian canvas made of ceramic with a smell of burning wood, an extra-extra-large German marker made of ceramic with a smell of popcorn, an extra-large Afghan hat made of plastic with a smell of popcorn, an extra-extra-small Spanish screwdriver made of steel with a smell of rose, a small Italian ring made of plastic with a smell of vanilla, an extra-extra-small Afghan apple made of ceramic with a smell of coconut, an extra-extra-large Italian pen made of steel with a smell of vanilla, a large French shoe made of steel with a smell of citrus fruits, an extra-small Afghan wallet made of glass with a smell of vanilla, an extra-extra-large Italian vase made of concrete with a smell of leather, an extra-small German bottle made of concrete with a smell of vinegar, a small Italian piano made of ceramic with a smell of citrus fruits, an extra-small Italian key made of glass with a smell of vinegar, an extra-extra-large Afghan jar made of concrete with a smell of wet dog, an extra-extra-small French scissors made of glass with a smell of leather, an extra-extra-small Japanese stapler made of glass with a smell of vinegar, a medium Afghan house made of plastic with a smell of gasoline, an extra-large German apple made of glass with a smell of vinegar, a medium French fork made of ceramic with a smell of wet dog, an extra-extra-large German sofa made of plastic with a smell of coconut, a medium Russian speaker made of plastic with a smell of coffee, an extra-large Japanese chair made of ceramic with a smell of burning wood, an extra-extra-small Canadian piano made of plastic with a smell of vanilla. The color of the items was respectively as follows: the first 1 was beige, the next 1 was black, the next 2 were blue, the next 1 was brown, the next 2 were crimson, the next 1 was cyan, the next 2 were gold, the next 1 was gray, the next 2 were green, the next 2 were ivory, the next 2 were khaki, the next 2 were magenta, the next 2 were maroon, the next 1 was orange, the next 3 were pink, the next 1 was red, the next 3 were silver, the next 2 were white, the next 2 were yellow. Then, for each item of size extra-large in my collection, my aunt took it but then gave me another one of the same item made of her favorite material and with a smell of citrus fruits (all other properties the same). After this, my collection had: 9 item(s) made of plastic, 10 item(s) made of concrete, 6 item(s) made of ceramic, 5 item(s) made of glass, 3 item(s) made of steel. Then, if I had an even number of an item type in my new collection (e.g. two knives, regardless of their size, color, origin, material, or smell), I threw all of them away. Then, my uncle threw away any Afghan item in my new collection, but gave me two of the same item, one German and one Russian (all other properties the same). Then, my brother replaced any item of size small in my new collection with an exact copy but with one specific size (different from small) and another exact copy but with color indigo. After this, my new collection had: 5 extra-small item(s), 5 extra-extra-small item(s), 3 large item(s), 7 medium item(s), 7 extra-extra-large item(s), 2 small item(s), 1 extra-large item(s). Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the extra-extra-small items in my collection.  In my current collection, how many items have the following attributes: either color is brown, or size is extra-extra-small, or material is plastic, or smell is vinegar, or origin is British? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}