{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"018d7cb3-9eb4-595a-bf07-93c4a18f2bfd","task_key":"bbeh~5fdyck~5flanguages--full--018d7cb3-9eb4-595a-bf07-93c4a18f2bfd","task_revision_id":"1","upstream_id":"","short_description":"You are an expert in a language called dyck where you must complete the language…","config":"bbeh_dyck_languages","split":"full","body":"{\"input\":\"You are an expert in a language called dyck where you must complete the language sequence of unclosed brackets of all types (e.g., [], {}, <>). You are given an initial initial dyck language sequence and the steps, provided as\\nthoughts, that were used to arrive at the closing bracket sequence in the dyck language. Your job is to identify the first step that was a mistake in reasoning about the closing bracket sequence in dyck language, if any.\\nThis can be forgetting to close a bracket or closing with the wrong bracket or incorrectly copying the prior subsequence of closing brackets to the next step.\\nTask: Complete the rest of the sequence, making sure that the parentheses are closed properly. Input: [ ( { < [ ] > < < [ { } ] > > } ( [ ( [ [ < ( ) > ] ] ( ( < [ < [ [ [ ] ] ] > ] > ) ) ) ] ) )\\nThought 1: We should process each input one by one and keep track of the stack configuration.\\nThought 2: stack: empty\\nThought 3: [ ; stack: [\\nThought 4: ( ; stack: [ (\\nThought 5: { ; stack: [ ( {\\nThought 6: < ; stack: [ ( { <\\nThought 7: [ ; stack: [ ( { < [\\nThought 8: ] ; stack: [ ( { <\\nThought 9: > ; stack: [ ( {\\nThought 10: < ; stack: [ ( { <\\nThought 11: [ ; stack: [ ( { < [\\nThought 12: { ; stack: [ ( { < [ {\\nThought 13: } ; stack: [ ( { < [\\nThought 14: ] ; stack: [ ( {\\nThought 15: > ; stack: [ (\\nThought 16: } ; stack: [\\nThought 17: ( ; stack: [ (\\nThought 18: [ ; stack: [ ( [\\nThought 19: [ ; stack: [ ( [ [\\nThought 20: [ ; stack: [ ( [ [ [\\nThought 21: < ; stack: [ ( [ [ [ <\\nThought 22: ( ; stack: [ ( [ [ [ < (\\nThought 23: ) ; stack: [ ( [ [ [ <\\nThought 24: > ; stack: [ ( [ [ [\\nThought 25: ] ; stack: [ ( [ [\\nThought 26: ] ; stack: [ ( [\\nThought 27: ( ; stack: [ ( [ (\\nThought 28: < ; stack: [ ( [ ( <\\nThought 29: [ ; stack: [ ( [ ( < [\\nThought 30: < ; stack: [ ( [ ( < [ <\\nThought 31: [ ; stack: [ ( [ ( < [ < [\\nThought 32: [ ; stack: [ ( [ ( < [ < [ [\\nThought 33: ] ; stack: [ ( [ ( < [ < [\\nThought 34: ] ; stack: [ ( [ ( < [ <\\nThought 35: ] ; stack: [ ( [ ( <\\nThought 36: > ; stack: [ ( [ (\\nThought 37: > ; stack: [ ( [\\nThought 38: ) ; stack: [ (\\nThought 39: ) ; stack: [\\nThought 40: ) ; stack: empty\\nThought 41: Now, we have reached the end. The final stack is empty.\\nThought 42: So the answer is ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) )\\nQ: Is there a mistake in this sequence? Write \\\"No\\\" if there are no mistakes, or the number N if there is a mistake in Thought N.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}