{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4781b46d-85b1-5b68-8f00-3a64c6f6e0b8","task_key":"bbeh~5fshuffled~5fobjects--full--4781b46d-85b1-5b68-8f00-3a64c6f6e0b8","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are friends and avid readers…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are friends and avid readers who occasionally trade books. At the start of the semester, they each buy one new book: Alice gets Catch-22, Bob gets The Fellowship of the Ring, Claire gets Frankenstein, Dave gets Moby Dick, Eve gets Ulysses, Fred gets The Pearl, and Gertrude gets The Odyssey.\\nAs the semester proceeds, they start trading around the new books. First, Alice and Eve swap books. Then, Dave and Eve swap books (let's call it Action 1). Then, Eve and Bob swap books (let's call it Action 2). Then, Bob and Eve swap books (let's call it Action 3). Then, Eve and Dave swap books (let's call it Action 4). Then, Dave and Bob swap books (let's call it Action 5). Then, Bob and Dave swap books (let's call it Action 6). Then, Bob and Alice swap books (let's call it Action 7). Then, Alice and Fred swap books (let's call it Action 8). Then, Fred and Alice swap books (let's call it Action 9). Then, Alice and Bob swap books (let's call it Action 10). Then, Claire and Alice swap books (let's call it Action 11). Then, Alice and Claire swap books (let's call it Action 12). Then, Gertrude and Claire swap books (let's call it Action 13). Then, Claire and Fred swap books (let's call it Action 14). Then, Fred and Claire swap books (let's call it Action 15). Then, Claire and Gertrude swap books (let's call it Action 16). Then, nothing happens for a while (let's call it Action 17). Then, Alice and Bob swap books. Then, Alice and Bob discuss something (let's call it Action 18). Then, Alice and Bob swap books. Then, Action 15 repeats. Then, Fred and Claire discuss something (let's call it Action 19). Then, Fred and Claire swap books. Then, Bob and Eve swap books. Then, Bob and Eve discuss something (let's call it Action 20). Then, Bob and Eve swap books. Then, Eve and Bob swap books. Then, Eve and Bob discuss something (let's call it Action 21). Then, Eve and Bob swap books. Then, Fred and Claire swap books. Then, Claire and Gertrude swap books. Then, Gertrude and Claire swap books. Then, Action 14 repeats. Then, Alice and Dave swap books (let's call it Action 22). Then, Dave and Fred swap books (let's call it Action 23). Then, Fred and Dave swap books (let's call it Action 24). Then, Dave and Alice swap books (let's call it Action 25). Then, nothing happens for a while. Then, Fred and Alice swap books. Then, Fred and Alice discuss something (let's call it Action 26). Then, Fred and Alice swap books. Then, Claire and Bob swap books (let's call it Action 27). Then, Claire and Bob discuss something (let's call it Action 28). Then, Claire and Bob swap books. Then, Eve and Dave swap books. Then, Dave and Claire swap books (let's call it Action 29). Then, Claire and Dave swap books (let's call it Action 30). Then, Dave and Eve swap books. Then, Eve and Gertrude swap books (let's call it Action 31). Then, Gertrude and Eve swap books (let's call it Action 32). Then, Gertrude and Bob swap books (let's call it Action 33). Then, Bob and Gertrude swap books (let's call it Action 34). Then, Gertrude and Bob swap books. Then, Bob and Gertrude swap books. Then, Bob and Eve swap books. Then, Bob and Eve discuss something. Then, Action 3 repeats. Then, Action 17 repeats. Then, Bob and Fred swap books (let's call it Action 35). Then, Bob and Fred discuss something (let's call it Action 36). Then, Bob and Fred swap books. Then, Claire and Gertrude swap books. Then, Claire and Gertrude discuss something (let's call it Action 37). Then, Claire and Gertrude swap books. Then, Bob and Eve swap books. Then, Bob and Eve discuss something. Then, Bob and Eve swap books. Then, nothing happens for a while. Then, Action 17 repeats. Then, Dave and Claire swap books. Then, Claire and Dave swap books. Then, nothing happens for a while. Then, Bob and Fred swap books. Then, Fred and Eve swap books (let's call it Action 38). Then, Eve and Fred swap books (let's call it Action 39). Then, Fred and Bob swap books (let's call it Action 40). Then, Claire and Bob swap books. Then, Bob and Claire swap books (let's call it Action 41). Then, Bob and Alice swap books. Then, Alice and Fred swap books. Then, Fred and Alice swap books. Then, Alice and Bob swap books. Then, nothing happens for a while. Then, Alice and Eve swap books (let's call it Action 42). Then, Eve and Fred swap books. Then, Fred and Eve swap books. Then, Eve and Alice swap books (let's call it Action 43). Then, nothing happens for a while. Then, Action 23 repeats. Then, Claire and Gertrude swap books. Then, Action 37 repeats. Then, Claire and Gertrude swap books. Then, Action 6 repeats. Then, Bob and Dave discuss something (let's call it Action 44). Then, Action 6 repeats. Then, Gertrude and Fred swap books (let's call it Action 45). Then, Fred and Bob swap books. Then, Bob and Fred swap books. Then, Fred and Gertrude swap books (let's call it Action 46). Then, Eve and Dave swap books. Then, Action 1 repeats. Then, nothing happens for a while. Then, Dave and Claire swap books. Then, Claire and Eve swap books (let's call it Action 47). Then, Eve and Claire swap books (let's call it Action 48). Then, Claire and Dave swap books. Then, Gertrude and Bob swap books. Then, Bob and Gertrude swap books. Then, Action 41 repeats. Then, Bob and Claire discuss something (let's call it Action 49). Then, Action 41 repeats. Then, Dave and Bob swap books. Then, Action 35 repeats. Then, Fred and Bob swap books. Then, Action 6 repeats. Then, Alice and Bob swap books. Then, Alice and Bob discuss something. Then, Action 10 repeats. Then, nothing happens for a while. Then, Dave and Eve swap books. Then, Eve and Dave swap books. Then, Gertrude and Bob swap books. Then, Bob and Dave swap books. Then, Dave and Bob swap books. Then, Bob and Gertrude swap books. Then, Alice and Fred swap books. Then, Alice and Fred discuss something (let's call it Action 50). Then, Alice and Fred swap books. Then, Dave and Gertrude swap books (let's call it Action 51). Then, Action 45 repeats. Then, Fred and Gertrude swap books. Then, Gertrude and Dave swap books (let's call it Action 52). Then, Dave and Fred swap books. Then, Fred and Dave swap books. Then, Bob and Eve swap books. Then, Eve and Claire swap books. Then, Claire and Eve swap books. Then, Eve and Bob swap books. Then, Bob and Alice swap books. Then, Alice and Gertrude swap books (let's call it Action 53). Then, Gertrude and Alice swap books (let's call it Action 54). Then, Alice and Bob swap books. Then, Fred and Alice swap books. Then, Fred and Alice discuss something. Then, Fred and Alice swap books. Then, nothing happens for a while. Then, Fred and Alice swap books. Then, Alice and Fred swap books. Then, Dave and Fred swap books. Then, Fred and Dave swap books. Then, Action 17 repeats. Then, Claire and Alice swap books. Then, Alice and Claire swap books. Then, Bob and Dave swap books. Then, Action 23 repeats. Then, Fred and Dave swap books. Then, Dave and Bob swap books. Then, Alice and Dave swap books. Then, Dave and Alice swap books. Then, Action 16 repeats. Then, Gertrude and Claire swap books. Then, Action 17 repeats. Then, Action 46 repeats. Then, Gertrude and Fred swap books. Then, nothing happens for a while. Then, Bob and Fred swap books. Then, Fred and Claire swap books. Then, Action 14 repeats. Then, Fred and Bob swap books. Then, Dave and Eve swap books. Then, Action 31 repeats. Then, Gertrude and Eve swap books. Then, Action 4 repeats. Then, Action 24 repeats. Then, Action 51 repeats. Then, Gertrude and Dave swap books. Then, Action 23 repeats. Then, Gertrude and Bob swap books. Then, nothing happens for a while. Then, Alice and Fred swap books. Then, Fred and Claire swap books. Then, Claire and Fred swap books. Then, Fred and Alice swap books. Then, Claire and Bob swap books. Then, Bob and Eve swap books. Then, Eve and Bob swap books. Then, Bob and Claire swap books. Then, Gertrude and Dave swap books. Then, Dave and Alice swap books. Then, Alice and Dave swap books. Then, Dave and Gertrude swap books. Then, Dave and Gertrude swap books. Then, Action 45 repeats. Then, Fred and Gertrude swap books. Then, Gertrude and Dave swap books. Then, Action 25 repeats. Then, Alice and Dave swap books. Then, Fred and Eve swap books. Then, Fred and Eve discuss something (let's call it Action 55). Then, Fred and Eve swap books. Then, nothing happens for a while. Then, Fred and Eve swap books. Then, Eve and Alice swap books. Then, Alice and Eve swap books. Then, Eve and Fred swap books. Then, Action 29 repeats. Then, Claire and Dave swap books. Then, Action 25 repeats. Then, Alice and Dave swap books. Then, Dave and Alice swap books. Then, Alice and Claire swap books. Then, Claire and Alice swap books. Then, Alice and Dave swap books. Then, Action 12 repeats. Then, Claire and Dave swap books. Then, Action 29 repeats. Then, Claire and Alice swap books. Then, Eve and Dave swap books. Then, Dave and Alice swap books. Then, Action 22 repeats. Then, Dave and Eve swap books. Then, nothing happens for a while. Then, Eve and Bob swap books. Then, Eve and Bob discuss something. Then, Eve and Bob swap books. Then, Fred and Dave swap books. Then, Dave and Fred swap books. Then, Claire and Dave swap books. Then, Dave and Claire swap books. Then, Bob and Eve swap books. Then, Action 43 repeats. Then, Alice and Eve swap books. Then, Eve and Bob swap books. Then, Gertrude and Eve swap books. Then, Eve and Alice swap books. Then, Alice and Eve swap books. Then, Eve and Gertrude swap books. Then, Alice and Claire swap books. Then, Claire and Dave swap books. Then, Dave and Claire swap books. Then, Action 11 repeats. Then, nothing happens for a while. Then, Alice and Claire swap books. Then, Bob and Eve swap books. Then, Eve and Bob swap books. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Action 12 repeats. Then, Claire and Eve swap books. Then, Eve and Claire swap books. Then, Claire and Alice swap books. Then, Bob and Dave swap books. Then, Action 5 repeats. Then, Claire and Eve swap books. Then, Eve and Claire swap books. Then, Action 17 repeats. Then, nothing happens for a while. Then, nothing happens for a while. Then, Action 10 repeats. Then, Bob and Claire swap books. Then, Claire and Bob swap books. Then, Bob and Alice swap books. Then, Bob and Dave swap books. Then, Action 29 repeats. Then, Claire and Dave swap books. Then, Dave and Bob swap books. Then, Fred and Dave swap books. Then, Dave and Fred swap books. Then, Claire and Bob swap books. Then, Bob and Claire swap books. Then, Gertrude and Claire swap books. Then, Action 16 repeats. Then, Dave and Claire swap books. Then, Claire and Dave swap books. Then, Action 6 repeats. Then, Bob and Dave discuss something. Then, Bob and Dave swap books. Then, Action 13 repeats. Then, Gertrude and Claire discuss something (let's call it Action 56). Then, Gertrude and Claire swap books. Then, Action 2 repeats. Then, Bob and Claire swap books. Then, Claire and Bob swap books. Then, Action 3 repeats. Then, Dave and Bob swap books. Then, Dave and Bob discuss something (let's call it Action 57). Then, Dave and Bob swap books. Then, Fred and Dave swap books. Then, Dave and Eve swap books. Then, Eve and Dave swap books. Then, Dave and Fred swap books. Then, Action 17 repeats. Then, Action 17 repeats. Then, Action 47 repeats. Then, Action 4 repeats. Then, Dave and Eve swap books. Then, Eve and Claire swap books. Then, Gertrude and Alice swap books. Then, Action 42 repeats. Then, Eve and Alice swap books. Then, Alice and Gertrude swap books. Then, Action 17 repeats. Then, Alice and Claire swap books. Then, Claire and Alice swap books. Then, Gertrude and Dave swap books. Then, Dave and Gertrude swap books. Then, Fred and Alice swap books. Then, Alice and Claire swap books. Then, Claire and Alice swap books. Then, Action 8 repeats. Then, Eve and Alice swap books. Then, Eve and Alice discuss something (let's call it Action 58). Then, Eve and Alice swap books. Then, Action 17 repeats. Then, Action 17 repeats. Then, Alice and Fred swap books. Then, Fred and Alice swap books. Then, Fred and Claire swap books. Then, Bob and Gertrude swap books. Finally, Claire and Fred swap books. At the end of the semester, Fred has\\nOptions:\\n(A) Catch-22\\n(B) The Fellowship of the Ring\\n(C) Frankenstein\\n(D) Moby Dick\\n(E) Ulysses\\n(F) The Pearl\\n(G) The Odyssey\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}