{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"eca086cf-8c64-5dd0-9805-47322e4c8a85","task_key":"bbeh~5fobject~5fproperties--full--eca086cf-8c64-5dd0-9805-47322e4c8a85","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 35 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 35 weird items that went through a few changes. Initially, I had a medium Spanish earring made of glass with a smell of coconut, an extra-extra-large Canadian drill made of glass with a smell of lavender, a medium Turkish umbrella made of steel with a smell of rose, a large British stapler made of glass with a smell of pine needles, a large French calculator made of concrete with a smell of gasoline, an extra-extra-small Spanish pen made of concrete with a smell of lavender, a medium Canadian bird made of concrete with a smell of vanilla, a large Canadian clock made of steel with a smell of popcorn, an extra-extra-large German chair made of steel with a smell of coffee, an extra-extra-large Russian bowl made of ceramic with a smell of burning wood, an extra-extra-small German plate made of steel with a smell of baking bread, an extra-extra-small Mexican cup made of glass with a smell of wet dog, an extra-large Russian box made of plastic with a smell of leather, an extra-extra-large German ring made of steel with a smell of lavender, an extra-small Iranian apple made of plastic with a smell of coffee, a medium Russian box made of glass with a smell of burning wood, a medium Polish saw made of plastic with a smell of lavender, an extra-extra-large Mexican camera made of glass with a smell of coffee, a small Polish key made of plastic with a smell of garlic, an extra-extra-large Japanese pencil made of concrete with a smell of rose, an extra-extra-small Polish drill made of glass with a smell of garlic, a large Polish box made of glass with a smell of lavender, an extra-extra-large Portuguese calculator made of steel with a smell of citrus fruits, an extra-extra-small Japanese ring made of concrete with a smell of chocolate, an extra-extra-small Turkish sunglasses made of concrete with a smell of freshly cut grass, a small Turkish necklace made of steel with a smell of garlic, a medium Japanese cup made of glass with a smell of citrus fruits, a large Brazilian clock made of steel with a smell of burning wood, an extra-large French plate made of glass with a smell of gasoline, an extra-extra-large Italian bicycle made of ceramic with a smell of baking bread, an extra-small Turkish shirt made of steel with a smell of chocolate, a large Spanish saw made of concrete with a smell of leather, an extra-large Japanese plate made of concrete with a smell of pine needles, an extra-large Canadian wallet made of concrete with a smell of leather, a medium British calculator made of plastic with a smell of citrus fruits. The color of the items was respectively as follows: the first 2 were beige, the next 1 was black, the next 4 were brown, the next 2 were crimson, the next 3 were gold, the next 1 was gray, the next 1 was indigo, the next 2 were magenta, the next 2 were maroon, the next 1 was orange, the next 4 were purple, the next 3 were red, the next 3 were silver, the next 2 were teal, the next 2 were turquoise, the next 2 were white. Then, for any item made of steel in my collection, my sister changed their color to her favorite color and changed their smell to wet dog. After this, my collection had: 2 beige item(s), 4 brown item(s), 2 gold item(s), 1 gray item(s), 2 magenta item(s), 2 maroon item(s), 1 orange item(s), 3 purple item(s), 2 red item(s), 2 silver item(s), 1 teal item(s), 2 turquoise item(s), 11 white item(s). Then, my brother replaced any item of size extra-small in my new collection with an exact copy but with one specific size (different from extra-small) and another exact copy but with color indigo. After this, my new collection had: 7 medium item(s), 8 extra-extra-large item(s), 8 large item(s), 6 extra-extra-small item(s), 4 extra-large item(s), 2 extra-small item(s), 2 small item(s). Then, my uncle threw away any British item in my new collection, but gave me two of the same item, one Iranian and one Turkish (all other properties the same). Then, my friend took any item with a smell of coffee, citrus fruits or pine needles in my new collection and threw it away. Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the small items in my collection.  In my current collection, how many items have the following attributes: either color is magenta, or size is small, or material is concrete, or smell is gasoline, or origin is Russian? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}