{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0224ea9a-c0a2-5fe3-845a-7e3bca8568ce","task_key":"bbeh~5fobject~5fproperties--full--0224ea9a-c0a2-5fe3-845a-7e3bca8568ce","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 31 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 31 weird items that went through a few changes. Initially, I had a small Brazilian pen made of plastic with a smell of gasoline, a small Canadian pen made of plastic with a smell of baking bread, an extra-extra-small Chinese shirt made of plastic with a smell of lavender, an extra-large Spanish vase made of ceramic with a smell of vanilla, a small Polish knife made of glass with a smell of gasoline, a medium Japanese sofa made of concrete with a smell of gasoline, an extra-extra-large British marker made of plastic with a smell of burning wood, an extra-large Chinese banana made of plastic with a smell of coffee, a large Spanish apple made of ceramic with a smell of burning wood, a medium Turkish fork made of plastic with a smell of freshly cut grass, an extra-large Mexican camera made of ceramic with a smell of vinegar, a large Italian bed made of plastic with a smell of popcorn, a small French flower pot made of plastic with a smell of rose, an extra-small German stapler made of steel with a smell of gasoline, a medium German sofa made of concrete with a smell of lavender, an extra-extra-large Portuguese trash can made of concrete with a smell of gasoline, an extra-large Canadian ruler made of concrete with a smell of gasoline, a medium Afghan sofa made of ceramic with a smell of garlic, an extra-extra-small Japanese dog made of concrete with a smell of baking bread, a medium German knife made of ceramic with a smell of citrus fruits, a small Canadian ball made of ceramic with a smell of rose, a large Portuguese dog made of ceramic with a smell of pine needles, an extra-small Brazilian dog made of glass with a smell of popcorn, an extra-extra-small Turkish bowl made of glass with a smell of citrus fruits, an extra-small Brazilian bed made of concrete with a smell of coconut, an extra-extra-large Japanese pencil made of ceramic with a smell of popcorn, an extra-large Canadian drill made of ceramic with a smell of popcorn, a large Polish bowl made of glass with a smell of rose, an extra-extra-large Portuguese candle made of ceramic with a smell of wet dog, an extra-extra-large German guitar made of plastic with a smell of chocolate, a large Russian plate made of ceramic with a smell of chocolate. The color of the items was respectively as follows: the first 3 were beige, the next 2 were black, the next 1 was blue, the next 5 were crimson, the next 1 was cyan, the next 3 were gold, the next 1 was gray, the next 1 was green, the next 1 was indigo, the next 3 were ivory, the next 3 were orange, the next 1 was pink, the next 1 was silver, the next 1 was turquoise, the next 2 were violet, the next 1 was white, the next 1 was yellow. Then, my friend took any item with a smell of wet dog, vinegar or baking bread in my collection and threw it away. Then, for each item of size medium in my new collection, my aunt took it but then gave me another one of the same item made of her favorite material and with a smell of pine needles (all other properties the same). After this, my new collection had: 7 item(s) made of plastic, 12 item(s) made of ceramic, 4 item(s) made of glass, 1 item(s) made of steel, 3 item(s) made of concrete. Then, if I had an even number of an item type in my new collection (e.g. two knives, regardless of their size, color, origin, material, or smell), I threw all of them away. Then, my uncle threw away any French item in my new collection, but gave me two of the same item, one Portuguese and one British (all other properties the same). Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the extra-extra-small items in my collection.  In my current collection, how many items have the following attributes: color is not indigo, size is not extra-extra-small, material is not steel, smell is not rose, and origin is not Turkish? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}