{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"124dc128-2c19-5281-a3b7-4d866292f7d1","task_key":"bbeh~5fobject~5fproperties--full--124dc128-2c19-5281-a3b7-4d866292f7d1","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 40 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 40 weird items that went through a few changes. Initially, I had an extra-small French banana made of glass with a smell of pine needles, a large Afghan table made of concrete with a smell of baking bread, a medium Mexican piano made of glass with a smell of wet dog, an extra-small Russian sofa made of glass with a smell of pine needles, an extra-extra-small British calculator made of glass with a smell of baking bread, an extra-extra-small Russian camera made of concrete with a smell of pine needles, a large Spanish vase made of plastic with a smell of freshly cut grass, a large Polish spoon made of steel with a smell of burning wood, a medium German calculator made of plastic with a smell of coffee, a medium British book made of plastic with a smell of rose, an extra-extra-large Afghan fork made of glass with a smell of chocolate, a small Turkish screwdriver made of steel with a smell of lavender, an extra-small Brazilian plier made of glass with a smell of wet dog, an extra-small Portuguese key made of glass with a smell of burning wood, a large Brazilian earring made of plastic with a smell of chocolate, an extra-extra-small Polish scissors made of ceramic with a smell of vinegar, a medium Mexican trash can made of concrete with a smell of leather, an extra-extra-small Polish wallet made of glass with a smell of vinegar, a medium British marker made of concrete with a smell of freshly cut grass, a large French shirt made of ceramic with a smell of gasoline, a medium Chinese fork made of concrete with a smell of burning wood, an extra-small Italian key made of steel with a smell of coconut, an extra-small Italian plate made of plastic with a smell of citrus fruits, an extra-large Brazilian key made of plastic with a smell of gasoline, a small Afghan flower pot made of concrete with a smell of citrus fruits, a large Japanese bottle made of plastic with a smell of leather, an extra-extra-large Polish wallet made of glass with a smell of popcorn, a large French sofa made of ceramic with a smell of citrus fruits, an extra-large Chinese calculator made of glass with a smell of gasoline, an extra-extra-small Japanese wallet made of ceramic with a smell of gasoline, an extra-small Mexican house made of concrete with a smell of freshly cut grass, a large German umbrella made of ceramic with a smell of coconut, a large Russian flower pot made of concrete with a smell of popcorn, an extra-extra-small Brazilian marker made of concrete with a smell of burning wood, a small British jar made of steel with a smell of popcorn, an extra-extra-small British box made of plastic with a smell of rose, an extra-extra-small Iranian watch made of steel with a smell of burning wood, a large Chinese chair made of ceramic with a smell of freshly cut grass, an extra-extra-large Russian pencil made of ceramic with a smell of lavender, a large French screwdriver made of ceramic with a smell of leather. The color of the items was respectively as follows: the first 1 was beige, the next 1 was blue, the next 3 were crimson, the next 1 was cyan, the next 3 were gold, the next 3 were gray, the next 2 were indigo, the next 1 was ivory, the next 1 was khaki, the next 2 were magenta, the next 4 were orange, the next 1 was pink, the next 4 were purple, the next 2 were red, the next 2 were silver, the next 2 were teal, the next 3 were turquoise, the next 1 was violet, the next 2 were white, the next 1 was yellow. Then, my brother replaced any item of size extra-small in my collection with an exact copy but with one specific size (different from extra-small) and another exact copy but with color gold. After this, my collection had: 13 medium item(s), 11 large item(s), 8 extra-extra-small item(s), 7 extra-small item(s), 3 extra-extra-large item(s), 3 small item(s), 2 extra-large item(s). Then, my uncle threw away any French item in my new collection, but gave me two of the same item, one Japanese and one Chinese (all other properties the same). Then, if I had an even number of an item type in my new collection (e.g. two knives, regardless of their size, color, origin, material, or smell), I threw all of them away. Then, my friend took any item with a smell of vinegar, coffee or popcorn in my new collection and threw it away. Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the extra-extra-small items in my collection.  In my current collection, how many items have the following attributes: color is not yellow, size is not extra-extra-small, material is not ceramic, smell is not gasoline, and origin is not Afghan? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}