{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ca15f47a-2e89-58c7-89ea-598c33e969f7","task_key":"bbeh~5fobject~5fproperties--full--ca15f47a-2e89-58c7-89ea-598c33e969f7","task_revision_id":"1","upstream_id":"","short_description":"I had a collection of 35 weird items that went through a few changes. Initially,…","config":"bbeh_object_properties","split":"full","body":"{\"input\":\"I had a collection of 35 weird items that went through a few changes. Initially, I had an extra-extra-large Japanese camera made of steel with a smell of citrus fruits, a medium Brazilian ruler made of ceramic with a smell of wet dog, an extra-large French dog made of ceramic with a smell of gasoline, an extra-extra-small French umbrella made of concrete with a smell of pine needles, an extra-extra-large Afghan fork made of plastic with a smell of gasoline, an extra-extra-large Afghan plier made of glass with a smell of vinegar, a large German bottle made of steel with a smell of pine needles, an extra-extra-large Mexican lamp made of ceramic with a smell of lavender, an extra-extra-large French plate made of concrete with a smell of chocolate, an extra-large Canadian calculator made of ceramic with a smell of wet dog, an extra-extra-large German marker made of ceramic with a smell of pine needles, a medium Brazilian ball made of plastic with a smell of vinegar, an extra-extra-large Mexican calculator made of glass with a smell of baking bread, an extra-small Chinese necklace made of glass with a smell of baking bread, a medium British camera made of steel with a smell of coconut, an extra-large Russian bowl made of concrete with a smell of lavender, an extra-extra-large Brazilian plate made of concrete with a smell of vinegar, an extra-small Spanish watch made of ceramic with a smell of vanilla, an extra-extra-large Iranian umbrella made of plastic with a smell of freshly cut grass, an extra-extra-large Japanese bed made of glass with a smell of wet dog, an extra-extra-large Brazilian wallet made of ceramic with a smell of pine needles, an extra-small Russian chair made of glass with a smell of coffee, a small Brazilian plate made of ceramic with a smell of garlic, an extra-large Spanish wrench made of plastic with a smell of leather, a medium Afghan speaker made of concrete with a smell of garlic, an extra-extra-large Spanish necklace made of glass with a smell of popcorn, a medium Russian bird made of plastic with a smell of freshly cut grass, a small British clock made of steel with a smell of coconut, a medium Canadian box made of ceramic with a smell of burning wood, an extra-small Portuguese jar made of glass with a smell of burning wood, an extra-small German stapler made of plastic with a smell of pine needles, a small Polish bird made of glass with a smell of coffee, a small British jar made of plastic with a smell of coconut, an extra-large Italian chair made of glass with a smell of popcorn, a large Portuguese screwdriver made of ceramic with a smell of vinegar. The color of the items was respectively as follows: the first 1 was beige, the next 1 was black, the next 2 were brown, the next 2 were crimson, the next 1 was cyan, the next 2 were gold, the next 4 were gray, the next 4 were green, the next 1 was indigo, the next 1 was ivory, the next 1 was khaki, the next 1 was magenta, the next 3 were orange, the next 1 was pink, the next 2 were purple, the next 2 were red, the next 1 was silver, the next 2 were teal, the next 2 were violet, the next 1 was yellow. Then, for any item of color pink in my collection, my mom gave me another one of the same item, but with a gray color, French origin, large size, citrus fruits smell, and made of concrete. Then, my friend took any item with a smell of chocolate, gasoline or leather in my new collection and threw it away. Then, for any item with a smell of wet dog, freshly cut grass, or coconut in my new collection, my cousin gifted me another one of that item with the same properties but with her favorite smell (different from the three smells above). After this, my new collection had: 2 item(s) with citrus fruits smell, 3 item(s) with wet dog smell, 8 item(s) with gasoline smell, 5 item(s) with pine needles smell, 4 item(s) with vinegar smell, 2 item(s) with lavender smell, 2 item(s) with baking bread smell, 3 item(s) with coconut smell, 1 item(s) with vanilla smell, 2 item(s) with freshly cut grass smell, 2 item(s) with coffee smell, 2 item(s) with garlic smell, 2 item(s) with popcorn smell, 2 item(s) with burning wood smell. Then, if I had an even number of an item type in my new collection (e.g. two knives, regardless of their size, color, origin, material, or smell), I threw all of them away. Then, my fiance compared my new collection with my initial collection and if there was an item in my new collection with a (smell, color) pair that was different from the (smell, color) pair of all items in my initial collection, she added another copy of that to my new collection.  Then, I lost one of the small items in my collection.  In my current collection, how many items have the following attributes: either color is magenta, or size is small, or material is concrete, or smell is lavender, or origin is Mexican? If the exact number cannot be computed, the answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}