{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1dd6e16f-5fa4-5a81-94cf-d7c838f4007b","task_key":"bbeh~5fobject~5fproperties--full--1dd6e16f-5fa4-5a81-94cf-d7c838f4007b","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 38 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 38 weird items that went through a few changes. Initially, I had an extra-extra-large French fork made of steel with a smell of gasoline, a medium Italian candle made of concrete with a smell of garlic, a large French wrench made of concrete with a smell of leather, a medium French book made of concrete with a smell of vanilla, an extra-extra-small Mexican brush made of ceramic with a smell of coffee, a small Polish canvas made of glass with a smell of freshly cut grass, a small French speaker made of plastic with a smell of wet dog, a large Iranian dog made of glass with a smell of rose, a small Japanese drill made of concrete with a smell of garlic, an extra-large Chinese lamp made of steel with a smell of chocolate, a large Spanish lamp made of glass with a smell of wet dog, a medium German plate made of plastic with a smell of coconut, a small Russian key made of plastic with a smell of leather, a small Iranian banana made of concrete with a smell of popcorn, a large Russian shirt made of glass with a smell of burning wood, an extra-large British apple made of steel with a smell of pine needles, a large Russian shoe made of ceramic with a smell of freshly cut grass, a small German plier made of plastic with a smell of vinegar, an extra-small Mexican pen made of plastic with a smell of burning wood, a large Canadian marker made of ceramic with a smell of popcorn, an extra-extra-small Brazilian canvas made of glass with a smell of popcorn, an extra-small Iranian knife made of plastic with a smell of coconut, an extra-extra-small Afghan marker made of ceramic with a smell of baking bread, an extra-small Portuguese car made of plastic with a smell of citrus fruits, an extra-large Afghan box made of ceramic with a smell of garlic, a large Canadian table made of glass with a smell of burning wood, an extra-extra-small Spanish bicycle made of steel with a smell of citrus fruits, an extra-extra-small Chinese wrench made of ceramic with a smell of vinegar, a large Polish jar made of plastic with a smell of popcorn, an extra-extra-small French cup made of plastic with a smell of coffee, an extra-extra-large French jar made of ceramic with a smell of popcorn, an extra-small Polish banana made of plastic with a smell of popcorn, a small French book made of ceramic with a smell of vanilla, an extra-small Italian bed made of glass with a smell of leather, a medium Brazilian marker made of plastic with a smell of rose, a large German earring made of steel with a smell of vinegar, an extra-large Portuguese house made of steel with a smell of lavender, an extra-large French stapler made of plastic with a smell of coconut. The color of the items was respectively as follows: the first 4 were beige, the next 3 were black, the next 1 was blue, the next 1 was crimson, the next 1 was cyan, the next 1 was gold, the next 2 were gray, the next 2 were indigo, the next 2 were ivory, the next 3 were khaki, the next 2 were magenta, the next 3 were maroon, the next 1 was orange, the next 1 was pink, the next 2 were red, the next 1 was silver, the next 1 was teal, the next 3 were turquoise, the next 2 were white, the next 2 were yellow. Then, for any item of color crimson in my collection, my mom gave me another one of the same item, but with a ivory color, Chinese origin, extra-extra-large size, gasoline smell, and made of ceramic. Then, my uncle threw away any Chinese item in my new collection, but gave me two of the same item, one Brazilian and one Polish (all other properties the same). Then, for each item of size extra-large in my new collection, my aunt took it but then gave me another one of the same item made of her favorite material and with a smell of leather (all other properties the same). After this, my new collection had: 3 item(s) made of steel, 5 item(s) made of concrete, 10 item(s) made of ceramic, 7 item(s) made of glass, 17 item(s) made of plastic. Then, my teacher took any item from my new collection whose color started with \\\"b\\\" or whose name had 6 characters and threw it away. Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the large items in my collection.  In my current collection, how many items have the following attributes: either color is gray, or size is large, or material is plastic, or smell is wet dog, or origin is Spanish? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}