{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0860c4c5-6ad6-5468-8c3d-c4a90255669f","task_key":"bbeh~5fobject~5fproperties--full--0860c4c5-6ad6-5468-8c3d-c4a90255669f","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 35 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 35 weird items that went through a few changes. Initially, I had an extra-large British house made of plastic with a smell of leather, a medium Polish marker made of ceramic with a smell of leather, a large German calculator made of steel with a smell of popcorn, an extra-extra-large Chinese speaker made of steel with a smell of leather, a medium Canadian jar made of glass with a smell of citrus fruits, an extra-extra-large Turkish bed made of steel with a smell of leather, an extra-small Canadian sunglasses made of concrete with a smell of lavender, an extra-small Polish screwdriver made of glass with a smell of freshly cut grass, an extra-extra-large Portuguese guitar made of glass with a smell of chocolate, an extra-extra-large Portuguese earring made of concrete with a smell of rose, a small Russian pencil made of plastic with a smell of chocolate, an extra-extra-small Mexican clock made of concrete with a smell of lavender, a medium Japanese pen made of plastic with a smell of vinegar, an extra-extra-large Afghan umbrella made of steel with a smell of pine needles, an extra-large Brazilian bicycle made of concrete with a smell of coffee, a medium Japanese chair made of glass with a smell of rose, a small Brazilian umbrella made of glass with a smell of baking bread, an extra-small Chinese clock made of plastic with a smell of coconut, an extra-large British earring made of concrete with a smell of rose, an extra-large German piano made of glass with a smell of wet dog, an extra-extra-small Mexican wallet made of plastic with a smell of coconut, a large Canadian plier made of plastic with a smell of chocolate, an extra-extra-large Italian knife made of plastic with a smell of burning wood, an extra-small French spoon made of steel with a smell of vanilla, an extra-large Afghan cup made of steel with a smell of burning wood, an extra-extra-large Iranian table made of plastic with a smell of chocolate, a large German shirt made of steel with a smell of vinegar, a large Turkish piano made of steel with a smell of garlic, a large Brazilian scissors made of concrete with a smell of leather, an extra-extra-small Italian wrench made of steel with a smell of coconut, a medium Polish calculator made of ceramic with a smell of gasoline, an extra-extra-small Iranian ruler made of steel with a smell of pine needles, an extra-extra-small German hammer made of glass with a smell of rose, an extra-large Chinese watch made of glass with a smell of pine needles, an extra-small Japanese hat made of concrete with a smell of baking bread. The color of the items was respectively as follows: the first 1 was beige, the next 3 were black, the next 1 was blue, the next 3 were crimson, the next 1 was gold, the next 5 were gray, the next 2 were green, the next 1 was indigo, the next 2 were ivory, the next 1 was khaki, the next 1 was magenta, the next 1 was maroon, the next 1 was pink, the next 4 were purple, the next 4 were silver, the next 1 was teal, the next 1 was white, the next 2 were yellow. Then, my teacher took any item from my collection whose color started with \\\"b\\\" or whose name had 6 characters and threw it away. Then, my friend took any item with a smell of chocolate, coffee or burning wood in my new collection and threw it away. Then, for each item of size medium in my new collection, my aunt took it but then gave me another one of the same item made of her favorite material and with a smell of vinegar (all other properties the same). After this, my new collection had: 9 item(s) made of steel, 6 item(s) made of concrete, 4 item(s) made of glass, 1 item(s) made of plastic. Then, for any item made of steel in my new collection, my sister changed their color to her favorite color and changed their smell to coconut. After this, my new collection had: 2 crimson item(s), 2 gray item(s), 1 indigo item(s), 2 ivory item(s), 10 khaki item(s), 1 silver item(s), 2 yellow item(s). Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the large items in my collection.  In my current collection, how many items have the following attributes: either color is gray, or size is large, or material is steel, or smell is rose, or origin is Portuguese? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}