{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0f30e3ec-2769-5b54-a197-944076cf2b69","task_key":"bbeh~5fshuffled~5fobjects--full--0f30e3ec-2769-5b54-a197-944076cf2b69","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are friends and avid readers…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are friends and avid readers who occasionally trade books. At the start of the semester, they each buy one new book: Alice gets Lolita, Bob gets The Great Gatsby, Claire gets The Fellowship of the Ring, Dave gets Catch-22, Eve gets The Odyssey, Fred gets Moby Dick, and Gertrude gets Ulysses.\\nAs the semester proceeds, they start trading around the new books. First, Eve and Gertrude swap books. Then, nothing happens for a while (let's call it Action 1). Then, nothing happens for a while. Then, Eve and Alice swap books (let's call it Action 2). Then, Alice and Eve swap books (let's call it Action 3). Then, Claire and Fred swap books (let's call it Action 4). Then, Fred and Alice swap books (let's call it Action 5). Then, Alice and Fred swap books (let's call it Action 6). Then, Fred and Claire swap books (let's call it Action 7). Then, Action 1 repeats. Then, Action 1 repeats. Then, Claire and Dave swap books (let's call it Action 8). Then, Dave and Gertrude swap books (let's call it Action 9). Then, Gertrude and Dave swap books (let's call it Action 10). Then, Dave and Claire swap books (let's call it Action 11). Then, nothing happens for a while. Then, Alice and Eve swap books. Then, Eve and Alice swap books. Then, nothing happens for a while. Then, Gertrude and Fred swap books (let's call it Action 12). Then, Fred and Eve swap books (let's call it Action 13). Then, Eve and Fred swap books (let's call it Action 14). Then, Fred and Gertrude swap books (let's call it Action 15). Then, nothing happens for a while. Then, Bob and Alice swap books (let's call it Action 16). Then, Alice and Bob swap books (let's call it Action 17). Then, Bob and Fred swap books (let's call it Action 18). Then, Fred and Gertrude swap books. Then, Action 12 repeats. Then, Fred and Bob swap books (let's call it Action 19). Then, Claire and Gertrude swap books (let's call it Action 20). Then, Gertrude and Claire swap books (let's call it Action 21). Then, Fred and Bob swap books. Then, Action 18 repeats. Then, Eve and Alice swap books. Then, Eve and Alice discuss something (let's call it Action 22). Then, Eve and Alice swap books. Then, Claire and Gertrude swap books. Then, Claire and Gertrude discuss something (let's call it Action 23). Then, Claire and Gertrude swap books. Then, Fred and Alice swap books. Then, Fred and Alice discuss something (let's call it Action 24). Then, Fred and Alice swap books. Then, Bob and Eve swap books (let's call it Action 25). Then, Eve and Fred swap books. Then, Fred and Eve swap books. Then, Eve and Bob swap books (let's call it Action 26). Then, nothing happens for a while. Then, Bob and Eve swap books. Then, nothing happens for a while. Then, Gertrude and Eve swap books (let's call it Action 27). Then, Eve and Fred swap books. Then, Action 13 repeats. Then, Eve and Gertrude swap books (let's call it Action 28). Then, Action 3 repeats. Then, Eve and Dave swap books (let's call it Action 29). Then, Dave and Eve swap books (let's call it Action 30). Then, Eve and Alice swap books. Then, nothing happens for a while. Then, Gertrude and Fred swap books. Then, Fred and Claire swap books. Then, Claire and Fred swap books. Then, Action 15 repeats. Then, Gertrude and Fred swap books. Then, Gertrude and Fred discuss something (let's call it Action 31). Then, Action 12 repeats. Then, Gertrude and Claire swap books. Then, Action 4 repeats. Then, Fred and Claire swap books. Then, Claire and Gertrude swap books. Then, Bob and Eve swap books. Then, Action 14 repeats. Then, Fred and Eve swap books. Then, Eve and Bob swap books. Then, nothing happens for a while. Then, Bob and Gertrude swap books (let's call it Action 32). Then, Bob and Gertrude discuss something (let's call it Action 33). Then, Bob and Gertrude swap books. Then, Action 13 repeats. Then, Action 2 repeats. Then, Action 3 repeats. Then, Eve and Fred swap books. Then, nothing happens for a while. Then, Action 1 repeats. Then, Action 18 repeats. Then, Fred and Eve swap books. Then, Action 14 repeats. Then, Fred and Bob swap books. Then, Action 20 repeats. Then, Claire and Gertrude discuss something. Then, Action 20 repeats. Then, Dave and Fred swap books (let's call it Action 34). Then, Dave and Fred discuss something (let's call it Action 35). Then, Action 34 repeats. Then, Fred and Bob swap books. Then, Bob and Eve swap books. Then, Eve and Bob swap books. Then, Bob and Fred swap books. Then, nothing happens for a while. Then, Dave and Claire swap books. Then, Dave and Claire discuss something (let's call it Action 36). Then, Dave and Claire swap books. Then, Gertrude and Fred swap books. Then, Fred and Dave swap books (let's call it Action 37). Then, Dave and Fred swap books. Then, Fred and Gertrude swap books. Then, Dave and Fred swap books. Then, Fred and Dave swap books. Then, nothing happens for a while. Then, Alice and Claire swap books (let's call it Action 38). Then, Alice and Claire discuss something (let's call it Action 39). Then, Alice and Claire swap books. Then, Action 27 repeats. Then, Eve and Fred swap books. Then, Fred and Eve swap books. Then, Eve and Gertrude swap books. Then, Gertrude and Alice swap books (let's call it Action 40). Then, Gertrude and Alice discuss something (let's call it Action 41). Then, Gertrude and Alice swap books. Then, Bob and Alice swap books. Then, Alice and Fred swap books. Then, Action 5 repeats. Then, Alice and Bob swap books. Then, Fred and Bob swap books. Then, Fred and Bob discuss something (let's call it Action 42). Then, Fred and Bob swap books. Then, Dave and Fred swap books. Then, Dave and Fred discuss something. Then, Dave and Fred swap books. Then, nothing happens for a while. Then, Eve and Fred swap books. Then, Action 7 repeats. Then, Action 4 repeats. Then, Fred and Eve swap books. Then, Eve and Alice swap books. Then, Alice and Eve swap books. Then, Claire and Fred swap books. Then, Action 7 repeats. Then, Bob and Dave swap books (let's call it Action 43). Then, Dave and Bob swap books (let's call it Action 44). Then, Action 38 repeats. Then, Claire and Alice swap books (let's call it Action 45). Then, Alice and Dave swap books (let's call it Action 46). Then, Dave and Alice swap books (let's call it Action 47). Then, nothing happens for a while. Then, Eve and Gertrude swap books. Then, Gertrude and Alice swap books. Then, Alice and Gertrude swap books (let's call it Action 48). Then, Gertrude and Eve swap books. Then, Eve and Fred swap books. Then, Eve and Fred discuss something (let's call it Action 49). Then, Eve and Fred swap books. Then, Action 32 repeats. Then, Gertrude and Dave swap books. Then, Action 9 repeats. Then, Gertrude and Bob swap books (let's call it Action 50). Then, nothing happens for a while. Then, Dave and Gertrude swap books. Then, Gertrude and Eve swap books. Then, Eve and Gertrude swap books. Then, Gertrude and Dave swap books. Then, Action 3 repeats. Then, Eve and Fred swap books. Then, Fred and Eve swap books. Then, Eve and Alice swap books. Then, Bob and Eve swap books. Then, Bob and Eve discuss something (let's call it Action 51). Then, Bob and Eve swap books. Then, Alice and Claire swap books. Then, Action 45 repeats. Then, Gertrude and Fred swap books. Then, Fred and Gertrude swap books. Then, Fred and Claire swap books. Then, Claire and Gertrude swap books. Then, Gertrude and Claire swap books. Then, Claire and Fred swap books. Then, Bob and Claire swap books (let's call it Action 52). Then, Bob and Claire discuss something (let's call it Action 53). Then, Bob and Claire swap books. Then, Gertrude and Dave swap books. Then, Dave and Gertrude swap books. Then, Fred and Gertrude swap books. Then, Gertrude and Dave swap books. Then, Dave and Gertrude swap books. Then, Gertrude and Fred swap books. Then, Action 48 repeats. Then, Action 16 repeats. Then, Alice and Bob swap books. Then, Action 1 repeats. Then, Eve and Fred swap books. Then, Fred and Claire swap books. Then, Claire and Fred swap books. Then, Fred and Eve swap books. Then, Fred and Alice swap books. Then, Alice and Gertrude swap books. Then, Action 40 repeats. Then, Alice and Fred swap books. Then, Alice and Fred swap books. Then, Fred and Dave swap books. Then, Dave and Fred swap books. Then, Fred and Alice swap books. Then, Action 37 repeats. Then, Action 47 repeats. Then, Alice and Dave swap books. Then, Dave and Fred swap books. Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Claire swap books. Then, Action 39 repeats. Then, Alice and Claire swap books. Then, Eve and Fred swap books. Then, Action 13 repeats. Then, Alice and Bob swap books. Then, Bob and Dave swap books. Then, Dave and Bob swap books. Then, Bob and Alice swap books. Then, Gertrude and Dave swap books. Then, Gertrude and Dave discuss something (let's call it Action 54). Then, Gertrude and Dave swap books. Then, Dave and Claire swap books. Then, Claire and Dave swap books. Then, Dave and Claire swap books. Then, Dave and Claire discuss something. Then, Action 11 repeats. Then, Dave and Eve swap books. Then, Eve and Dave swap books. Then, Eve and Dave swap books. Then, Dave and Eve swap books. Then, Eve and Bob swap books. Then, Action 32 repeats. Then, Action 50 repeats. Then, Bob and Eve swap books. Then, Fred and Bob swap books. Then, Fred and Bob discuss something. Then, Fred and Bob swap books. Then, Action 47 repeats. Then, Alice and Fred swap books. Then, Fred and Alice swap books. Then, Alice and Dave swap books. Then, nothing happens for a while. Then, Alice and Eve swap books. Then, Eve and Gertrude swap books. Then, Gertrude and Eve swap books. Then, Eve and Alice swap books. Then, Alice and Claire swap books. Then, Action 45 repeats. Then, Claire and Eve swap books (let's call it Action 55). Then, nothing happens for a while. Then, Gertrude and Dave swap books. Then, Gertrude and Dave discuss something. Then, Gertrude and Dave swap books. Then, Gertrude and Fred swap books. Then, Fred and Gertrude swap books. Then, Bob and Fred swap books. Then, Fred and Dave swap books. Then, Dave and Fred swap books. Then, Action 19 repeats. Then, Eve and Claire swap books (let's call it Action 56). Then, Claire and Eve swap books. Then, Action 11 repeats. Then, Claire and Dave swap books. Then, Gertrude and Bob swap books. Then, Gertrude and Bob discuss something (let's call it Action 57). Then, Gertrude and Bob swap books. Then, Bob and Eve swap books. Then, Eve and Bob swap books. Then, Alice and Fred swap books. Then, Fred and Alice swap books. Then, Gertrude and Eve swap books. Then, Gertrude and Eve discuss something (let's call it Action 58). Then, Gertrude and Eve swap books. Then, Action 1 repeats. Then, Dave and Fred swap books. Then, Dave and Fred discuss something. Then, Dave and Fred swap books. Then, Gertrude and Eve swap books. Then, Action 58 repeats. Then, Gertrude and Eve swap books. Then, Fred and Bob swap books. Then, Action 42 repeats. Then, Action 19 repeats. Then, nothing happens for a while. Then, nothing happens for a while. Then, Eve and Bob swap books. Then, Bob and Eve swap books. Then, Alice and Claire swap books. Then, Alice and Claire discuss something. Then, Alice and Claire swap books. Then, Eve and Claire swap books. Then, Eve and Claire discuss something (let's call it Action 59). Then, Action 56 repeats. Then, Eve and Alice swap books. Then, Alice and Fred swap books. Then, Fred and Alice swap books. Then, Alice and Eve swap books. Then, Action 6 repeats. Then, Action 5 repeats. Then, Action 1 repeats. Then, nothing happens for a while. Then, Dave and Eve swap books. Then, Eve and Dave swap books. Then, Dave and Eve swap books. Then, Eve and Dave swap books. Then, Action 1 repeats. Then, Gertrude and Claire swap books. Then, Claire and Gertrude swap books. Then, nothing happens for a while. Then, Eve and Alice swap books. Then, Fred and Alice swap books. Finally, Eve and Dave swap books. At the end of the semester, Dave has\\nOptions:\\n(A) Lolita\\n(B) The Great Gatsby\\n(C) The Fellowship of the Ring\\n(D) Catch-22\\n(E) The Odyssey\\n(F) Moby Dick\\n(G) Ulysses\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}