{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"38c7ae53-43ad-51a9-94ac-9b12d1bad284","task_key":"bbeh~5fshuffled~5fobjects--full--38c7ae53-43ad-51a9-94ac-9b12d1bad284","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are friends and avid readers…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are friends and avid readers who occasionally trade books. At the start of the semester, they each buy one new book: Alice gets Moby Dick, Bob gets The Odyssey, Claire gets Ulysses, Dave gets The Fellowship of the Ring, Eve gets Lolita, Fred gets The Pearl, and Gertrude gets Catch-22.\\nAs the semester proceeds, they start trading around the new books. First, Dave and Claire swap books. Then, nothing happens for a while (let's call it Action 1). Then, Alice and Dave swap books (let's call it Action 2). Then, Alice and Dave discuss something (let's call it Action 3). Then, Action 2 repeats. Then, nothing happens for a while. Then, nothing happens for a while. Then, Fred and Claire swap books (let's call it Action 4). Then, Fred and Claire discuss something (let's call it Action 5). Then, Fred and Claire swap books. Then, Claire and Bob swap books (let's call it Action 6). Then, Bob and Eve swap books (let's call it Action 7). Then, Eve and Bob swap books (let's call it Action 8). Then, Bob and Claire swap books (let's call it Action 9). Then, Action 1 repeats. Then, Action 1 repeats. Then, Dave and Alice swap books (let's call it Action 10). Then, Alice and Dave swap books. Then, Claire and Gertrude swap books (let's call it Action 11). Then, Gertrude and Eve swap books (let's call it Action 12). Then, Eve and Gertrude swap books (let's call it Action 13). Then, Gertrude and Claire swap books (let's call it Action 14). Then, Action 1 repeats. Then, Claire and Alice swap books (let's call it Action 15). Then, Alice and Dave swap books. Then, Dave and Alice swap books. Then, Alice and Claire swap books (let's call it Action 16). Then, Action 2 repeats. Then, Action 10 repeats. Then, Action 12 repeats. Then, Eve and Gertrude swap books. Then, nothing happens for a while. Then, Eve and Dave swap books (let's call it Action 17). Then, Dave and Eve swap books (let's call it Action 18). Then, nothing happens for a while. Then, Eve and Alice swap books (let's call it Action 19). Then, Alice and Eve swap books (let's call it Action 20). Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Dave swap books. Then, Action 10 repeats. Then, Fred and Dave swap books (let's call it Action 21). Then, Dave and Fred swap books (let's call it Action 22). Then, Gertrude and Eve swap books. Then, Gertrude and Eve discuss something (let's call it Action 23). Then, Gertrude and Eve swap books. Then, Fred and Claire swap books. Then, Claire and Dave swap books (let's call it Action 24). Then, Dave and Claire swap books (let's call it Action 25). Then, Claire and Fred swap books (let's call it Action 26). Then, Gertrude and Fred swap books (let's call it Action 27). Then, Fred and Claire swap books. Then, Claire and Fred swap books. Then, Fred and Gertrude swap books (let's call it Action 28). Then, Action 27 repeats. Then, nothing happens for a while. Then, Bob and Fred swap books (let's call it Action 29). Then, Fred and Bob swap books (let's call it Action 30). Then, Dave and Bob swap books (let's call it Action 31). Then, Bob and Alice swap books (let's call it Action 32). Then, Alice and Bob swap books (let's call it Action 33). Then, Bob and Dave swap books (let's call it Action 34). Then, Alice and Bob swap books. Then, Alice and Bob discuss something (let's call it Action 35). Then, Alice and Bob swap books. Then, Action 9 repeats. Then, Claire and Bob swap books. Then, Bob and Claire swap books. Then, Action 6 repeats. Then, Fred and Gertrude swap books. Then, Gertrude and Dave swap books (let's call it Action 36). Then, Dave and Gertrude swap books (let's call it Action 37). Then, Gertrude and Fred swap books. Then, Alice and Gertrude swap books (let's call it Action 38). Then, Alice and Gertrude discuss something (let's call it Action 39). Then, Alice and Gertrude swap books. Then, Action 9 repeats. Then, Claire and Bob swap books. Then, Dave and Gertrude swap books. Then, Gertrude and Bob swap books (let's call it Action 40). Then, Bob and Gertrude swap books (let's call it Action 41). Then, Gertrude and Dave swap books. Then, Eve and Bob swap books. Then, Action 29 repeats. Then, Fred and Bob swap books. Then, Bob and Eve swap books. Then, Fred and Bob swap books. Then, Bob and Eve swap books. Then, Eve and Bob swap books. Then, Bob and Fred swap books. Then, Bob and Dave swap books. Then, Bob and Dave discuss something (let's call it Action 42). Then, Action 34 repeats. Then, Action 17 repeats. Then, Action 31 repeats. Then, Bob and Dave swap books. Then, Dave and Eve swap books. Then, Alice and Claire swap books. Then, Claire and Fred swap books. Then, Fred and Claire swap books. Then, Action 15 repeats. Then, Action 1 repeats. Then, nothing happens for a while. Then, nothing happens for a while. Then, Eve and Bob swap books. Then, Bob and Eve swap books. Then, nothing happens for a while. Then, Action 1 repeats. Then, Alice and Bob swap books. Then, Action 34 repeats. Then, Action 31 repeats. Then, Bob and Alice swap books. Then, Bob and Eve swap books. Then, Action 8 repeats. Then, Gertrude and Dave swap books. Then, Dave and Gertrude swap books. Then, Action 19 repeats. Then, nothing happens for a while. Then, Action 33 repeats. Then, Action 34 repeats. Then, Dave and Bob swap books. Then, Bob and Alice swap books. Then, Action 1 repeats. Then, Bob and Eve swap books. Then, Bob and Eve discuss something (let's call it Action 43). Then, Bob and Eve swap books. Then, Action 17 repeats. Then, Eve and Dave discuss something (let's call it Action 44). Then, Eve and Dave swap books. Then, Action 12 repeats. Then, Eve and Dave swap books. Then, Dave and Eve swap books. Then, Action 13 repeats. Then, Fred and Gertrude swap books. Then, Fred and Gertrude discuss something (let's call it Action 45). Then, Fred and Gertrude swap books. Then, Action 40 repeats. Then, Action 29 repeats. Then, Action 30 repeats. Then, Bob and Gertrude swap books. Then, nothing happens for a while. Then, Eve and Dave swap books. Then, Action 44 repeats. Then, Eve and Dave swap books. Then, Gertrude and Claire swap books. Then, Action 15 repeats. Then, Alice and Claire swap books. Then, Claire and Gertrude swap books. Then, Bob and Dave swap books. Then, Bob and Dave discuss something. Then, Bob and Dave swap books. Then, Action 27 repeats. Then, Fred and Gertrude swap books. Then, Bob and Gertrude swap books. Then, Gertrude and Bob swap books. Then, Gertrude and Alice swap books (let's call it Action 46). Then, Alice and Gertrude swap books. Then, Action 20 repeats. Then, Eve and Dave swap books. Then, Action 18 repeats. Then, Eve and Alice swap books. Then, Action 46 repeats. Then, Alice and Gertrude swap books. Then, nothing happens for a while. Then, Dave and Eve swap books. Then, Eve and Dave swap books. Then, Dave and Alice swap books. Then, Alice and Dave swap books. Then, Eve and Fred swap books (let's call it Action 47). Then, Action 28 repeats. Then, Gertrude and Fred swap books. Then, Fred and Eve swap books (let's call it Action 48). Then, Dave and Alice swap books. Then, Claire and Dave swap books. Then, Action 25 repeats. Then, Eve and Bob swap books. Then, Action 41 repeats. Then, Gertrude and Bob swap books. Then, Bob and Eve swap books. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Action 41 repeats. Then, Action 36 repeats. Then, Dave and Gertrude swap books. Then, Action 40 repeats. Then, nothing happens for a while. Then, Dave and Gertrude swap books. Then, Gertrude and Dave swap books. Then, nothing happens for a while. Then, nothing happens for a while. Then, Fred and Claire swap books. Then, Action 26 repeats. Then, Action 28 repeats. Then, Gertrude and Fred swap books. Then, Eve and Bob swap books. Then, Eve and Bob discuss something (let's call it Action 49). Then, Action 8 repeats. Then, Eve and Dave swap books. Then, Dave and Alice swap books. Then, Alice and Dave swap books. Then, Action 18 repeats. Then, Eve and Claire swap books (let's call it Action 50). Then, Claire and Alice swap books. Then, Alice and Claire swap books. Then, Claire and Eve swap books (let's call it Action 51). Then, Eve and Bob swap books. Then, Action 49 repeats. Then, Eve and Bob swap books. Then, Bob and Alice swap books. Then, Action 2 repeats. Then, Dave and Alice swap books. Then, Action 33 repeats. Then, Action 1 repeats. Then, Alice and Claire swap books. Then, Claire and Bob swap books. Then, Action 9 repeats. Then, Claire and Alice swap books. Then, Action 2 repeats. Then, Action 3 repeats. Then, Alice and Dave swap books. Then, Gertrude and Fred swap books. Then, Gertrude and Fred discuss something (let's call it Action 52). Then, Gertrude and Fred swap books. Then, Bob and Claire swap books. Then, Bob and Claire discuss something (let's call it Action 53). Then, Bob and Claire swap books. Then, Claire and Dave swap books. Then, Claire and Dave discuss something (let's call it Action 54). Then, Claire and Dave swap books. Then, Dave and Fred swap books. Then, Fred and Eve swap books. Then, Eve and Fred swap books. Then, Fred and Dave swap books. Then, Gertrude and Claire swap books. Then, Gertrude and Claire discuss something (let's call it Action 55). Then, Action 14 repeats. Then, Action 18 repeats. Then, Eve and Dave swap books. Then, Eve and Fred swap books. Then, Fred and Alice swap books (let's call it Action 56). Then, Alice and Fred swap books (let's call it Action 57). Then, Fred and Eve swap books. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Gertrude swap books. Then, Gertrude and Dave swap books. Then, Claire and Alice swap books. Then, Claire and Alice discuss something (let's call it Action 58). Then, Claire and Alice swap books. Then, Eve and Dave swap books. Then, Eve and Dave discuss something. Then, Action 17 repeats. Then, Action 28 repeats. Then, Gertrude and Fred swap books. Then, Action 33 repeats. Then, Action 35 repeats. Then, Alice and Bob swap books. Then, Dave and Claire swap books. Then, Claire and Dave swap books. Then, Action 1 repeats. Then, nothing happens for a while. Then, Eve and Claire swap books. Then, Eve and Claire discuss something (let's call it Action 59). Then, Action 50 repeats. Then, Eve and Alice swap books. Then, Alice and Eve swap books. Then, Gertrude and Claire swap books. Then, Gertrude and Bob swap books. Finally, Bob and Eve swap books. At the end of the semester, Claire has\\nOptions:\\n(A) Moby Dick\\n(B) The Odyssey\\n(C) Ulysses\\n(D) The Fellowship of the Ring\\n(E) Lolita\\n(F) The Pearl\\n(G) Catch-22\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}