{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7a75aa7a-af78-5707-9cb9-ff1ef6507c47","task_key":"bbeh~5fobject~5fproperties--full--7a75aa7a-af78-5707-9cb9-ff1ef6507c47","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 31 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 31 weird items that went through a few changes. Initially, I had a small Canadian chair made of plastic with a smell of vanilla, a large Japanese bowl made of plastic with a smell of coconut, an extra-large Mexican guitar made of glass with a smell of baking bread, a large Chinese earring made of glass with a smell of chocolate, an extra-large Portuguese car made of glass with a smell of leather, a small Portuguese candle made of plastic with a smell of pine needles, an extra-extra-large Mexican camera made of plastic with a smell of chocolate, an extra-large Canadian key made of concrete with a smell of freshly cut grass, an extra-extra-small Japanese trash can made of ceramic with a smell of baking bread, a small British hat made of steel with a smell of burning wood, an extra-extra-small Japanese speaker made of concrete with a smell of lavender, a small Russian pen made of concrete with a smell of popcorn, a small Afghan flower pot made of ceramic with a smell of rose, an extra-large Afghan brush made of ceramic with a smell of gasoline, an extra-extra-small Russian hat made of ceramic with a smell of gasoline, an extra-extra-large Polish bed made of steel with a smell of lavender, an extra-extra-small French brush made of glass with a smell of vanilla, a large Japanese hammer made of ceramic with a smell of burning wood, a small British dog made of steel with a smell of vanilla, an extra-extra-large Polish wrench made of glass with a smell of lavender, an extra-large Mexican shoe made of glass with a smell of burning wood, a small Afghan ring made of plastic with a smell of vinegar, an extra-large Russian clock made of glass with a smell of leather, an extra-extra-small Mexican ruler made of plastic with a smell of vinegar, an extra-extra-large Mexican book made of plastic with a smell of coffee, an extra-small Portuguese brush made of steel with a smell of vinegar, a large Canadian table made of ceramic with a smell of coconut, an extra-small Brazilian box made of ceramic with a smell of burning wood, an extra-extra-large Italian dog made of ceramic with a smell of vanilla, an extra-large Japanese calculator made of plastic with a smell of freshly cut grass, a medium Brazilian bed made of concrete with a smell of wet dog. The color of the items was respectively as follows: the first 3 were black, the next 1 was blue, the next 2 were brown, the next 1 was crimson, the next 4 were cyan, the next 2 were gray, the next 1 was indigo, the next 5 were ivory, the next 3 were maroon, the next 3 were orange, the next 2 were pink, the next 1 was teal, the next 2 were violet, the next 1 was yellow. Then, my friend took any item with a smell of vanilla, gasoline or rose in my collection and threw it away. Then, for any item with a smell of chocolate, vinegar, or wet dog in my new collection, my cousin gifted me another one of that item with the same properties but with her favorite smell (different from the three smells above). After this, my new collection had: 8 item(s) with coconut smell, 2 item(s) with baking bread smell, 2 item(s) with chocolate smell, 2 item(s) with leather smell, 1 item(s) with pine needles smell, 2 item(s) with freshly cut grass smell, 4 item(s) with burning wood smell, 3 item(s) with lavender smell, 1 item(s) with popcorn smell, 3 item(s) with vinegar smell, 1 item(s) with coffee smell, 1 item(s) with wet dog smell. Then, if I had an even number of an item type in my new collection (e.g. two knives, regardless of their size, color, origin, material, or smell), I threw all of them away. Then, my uncle threw away any Portuguese item in my new collection, but gave me two of the same item, one British and one Brazilian (all other properties the same). Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the large items in my collection.  In my current collection, how many items have the following attributes: either color is ivory, or size is large, or material is concrete, or smell is coconut, or origin is Brazilian? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}