{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"dae02940-4f98-5c7f-afe5-0ac71e43e520","task_key":"bbeh~5fword~5fsorting--full--dae02940-4f98-5c7f-afe5-0ac71e43e520","task_revision_id":"1","upstream_id":"","short_description":"You are an expert in word sorting. You will be provided with a list of words and…","config":"bbeh_word_sorting","split":"full","body":"{\"input\":\"\\nYou are an expert in word sorting. You will be provided with a list of words and the steps taken, in the form of thoughts, to arrange them in alphabetical order.\\nYour job is to identify the first step that was a mistake in reasoning about the order of the words. This can be misunderstanding the order of\\nthe alphabet or getting the word or letter wrong or missing a word entirely.\\nQ: Sort the following words alphabetically: List: celebes specimen krakow coleman topmost vaughn hagen whop cookie blurb tonk crosspoint sagacity course hattie zoom wanton tarpaper rd\\nThought 1: I should start by looking at the first letter of the words in the list. The first letter: \\\"celebes\\\": \\\"c\\\" (3). \\\"specimen\\\": \\\"s\\\" (19). \\\"krakow\\\": \\\"k\\\" (11). \\\"coleman\\\": \\\"c\\\" (3). \\\"topmost\\\": \\\"t\\\" (20). \\\"vaughn\\\": \\\"v\\\" (22). \\\"hagen\\\": \\\"h\\\" (8). \\\"whop\\\": \\\"w\\\" (23). \\\"cookie\\\": \\\"c\\\" (3). \\\"blurb\\\": \\\"b\\\" (2). \\\"tonk\\\": \\\"t\\\" (20). \\\"crosspoint\\\": \\\"c\\\" (3). \\\"sagacity\\\": \\\"s\\\" (19). \\\"course\\\": \\\"c\\\" (3). \\\"hattie\\\": \\\"h\\\" (8). \\\"zoom\\\": \\\"z\\\" (26). \\\"wanton\\\": \\\"w\\\" (23). \\\"tarpaper\\\": \\\"t\\\" (20). \\\"rd\\\": \\\"r\\\" (18).\\nThought 2: We now have: (2) \\\"blurb\\\" < (3) [\\\"celebes\\\" ? \\\"coleman\\\" ? \\\"cookie\\\" ? \\\"crosspoint\\\" ? \\\"course\\\"] < (8) [\\\"hagen\\\" ? \\\"hattie\\\"] < (11) \\\"krakow\\\" < (18) \\\"rd\\\" < (19) [\\\"sagacity\\\" ? \\\"specimen\\\"] < (20) [\\\"topmost\\\" ? \\\"tonk\\\" ? \\\"tarpaper\\\"] < (22) \\\"vaughn\\\" < (23) [\\\"wanton\\\" ? \\\"whop\\\"] < (26) \\\"zoom\\\".\\nThought 3: Now let's sort this subpart [\\\"celebes\\\" ? \\\"coleman\\\" ? \\\"cookie\\\" ? \\\"crosspoint\\\" ? \\\"course\\\"] by looking at their second letters. The second letter: \\\"celebes\\\": \\\"e\\\" (5). \\\"coleman\\\": \\\"o\\\" (15). \\\"cookie\\\": \\\"o\\\" (15). \\\"crosspoint\\\": \\\"r\\\" (18). \\\"course\\\": \\\"o\\\" (15).\\nThought 4: We now have: (5) \\\"celebes\\\" < (15) [\\\"coleman\\\" ? \\\"cookie\\\" ? \\\"course\\\"] < (18) \\\"crosspoint\\\" for the subpart.\\nThought 5: Now let's sort this subpart [\\\"coleman\\\" ? \\\"cookie\\\" ? \\\"course\\\"] by looking at their third letters. The third letter: \\\"coleman\\\": \\\"e\\\" (5). \\\"cookie\\\": \\\"k\\\" (11). \\\"course\\\": \\\"u\\\" (21).\\nThought 6: We now have: (5) \\\"coleman\\\" < (11) \\\"cookie\\\" < (21) \\\"course\\\" for the subpart. Hence, we have \\\"blurb\\\" < \\\"celebes\\\" < \\\"coleman\\\" < \\\"cookie\\\" < \\\"crosspoint\\\" < \\\"course\\\" < \\\"hagen\\\" < \\\"hattie\\\" < \\\"krakow\\\" < \\\"rd\\\" < \\\"sagacity\\\" < \\\"specimen\\\" < \\\"topmost\\\" < \\\"tonk\\\" < \\\"tarpaper\\\" < \\\"vaughn\\\" < \\\"wanton\\\" < \\\"whop\\\" < \\\"zoom\\\".\\nThought 7: I have now sorted all the words. The answer is blurb celebes coleman cookie crosspoint course hagen hattie krakow rd sagacity specimen topmost tonk tarpaper vaughn wanton whop zoom\\nQ: Is there a mistake in this sequence? Write \\\"No\\\" if there are no mistakes, or the number N if there is a mistake in Thought N.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}