{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"16966de0-682d-5eb5-939f-25a84eabbcb6","task_key":"bbeh~5fshuffled~5fobjects--full--16966de0-682d-5eb5-939f-25a84eabbcb6","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are friends and avid readers…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are friends and avid readers who occasionally trade books. At the start of the semester, they each buy one new book: Alice gets The Great Gatsby, Bob gets Catch-22, Claire gets The Fellowship of the Ring, Dave gets Ulysses, Eve gets Frankenstein, Fred gets The Pearl, and Gertrude gets The Odyssey.\\nAs the semester proceeds, they start trading around the new books. First, Fred and Alice swap books. Then, Dave and Gertrude swap books (let's call it Action 1). Then, Gertrude and Dave swap books (let's call it Action 2). Then, Claire and Dave swap books (let's call it Action 3). Then, Claire and Dave discuss something (let's call it Action 4). Then, Claire and Dave swap books. Then, nothing happens for a while (let's call it Action 5). Then, Claire and Dave swap books. Then, Claire and Dave discuss something. Then, Claire and Dave swap books. Then, Fred and Bob swap books (let's call it Action 6). Then, Bob and Fred swap books (let's call it Action 7). Then, Gertrude and Fred swap books (let's call it Action 8). Then, Fred and Gertrude swap books (let's call it Action 9). Then, nothing happens for a while. Then, Claire and Eve swap books (let's call it Action 10). Then, Eve and Claire swap books (let's call it Action 11). Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Bob swap books (let's call it Action 12). Then, Bob and Alice swap books (let's call it Action 13). Then, Alice and Dave swap books (let's call it Action 14). Then, Alice and Dave discuss something (let's call it Action 15). Then, Action 14 repeats. Then, Bob and Gertrude swap books (let's call it Action 16). Then, Gertrude and Bob swap books (let's call it Action 17). Then, Dave and Bob swap books (let's call it Action 18). Then, Bob and Dave swap books (let's call it Action 19). Then, Action 5 repeats. Then, Gertrude and Dave swap books. Then, Gertrude and Dave discuss something (let's call it Action 20). Then, Action 2 repeats. Then, nothing happens for a while. Then, Dave and Eve swap books (let's call it Action 21). Then, Dave and Eve discuss something (let's call it Action 22). Then, Dave and Eve swap books. Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Gertrude swap books (let's call it Action 23). Then, Gertrude and Bob swap books. Then, Bob and Gertrude swap books. Then, Gertrude and Alice swap books (let's call it Action 24). Then, Dave and Fred swap books (let's call it Action 25). Then, Fred and Alice swap books (let's call it Action 26). Then, Alice and Fred swap books (let's call it Action 27). Then, Fred and Dave swap books (let's call it Action 28). Then, Alice and Claire swap books (let's call it Action 29). Then, Claire and Gertrude swap books (let's call it Action 30). Then, Gertrude and Claire swap books (let's call it Action 31). Then, Claire and Alice swap books (let's call it Action 32). Then, Fred and Gertrude swap books. Then, Gertrude and Fred swap books. Then, Fred and Alice swap books. Then, Alice and Bob swap books. Then, Bob and Alice swap books. Then, Action 27 repeats. Then, Action 26 repeats. Then, Action 14 repeats. Then, Dave and Alice swap books (let's call it Action 33). Then, Alice and Fred swap books. Then, Action 27 repeats. Then, Fred and Alice swap books. Then, Bob and Gertrude swap books. Then, Gertrude and Eve swap books (let's call it Action 34). Then, Eve and Gertrude swap books (let's call it Action 35). Then, Gertrude and Bob swap books. Then, Action 5 repeats. Then, Eve and Bob swap books (let's call it Action 36). Then, Bob and Alice swap books. Then, Action 12 repeats. Then, Bob and Eve swap books (let's call it Action 37). Then, Claire and Dave swap books. Then, Dave and Claire swap books (let's call it Action 38). Then, nothing happens for a while. Then, Action 34 repeats. Then, Gertrude and Bob swap books. Then, Bob and Fred swap books. Then, Fred and Bob swap books. Then, Action 16 repeats. Then, Gertrude and Fred swap books. Then, Gertrude and Fred discuss something (let's call it Action 39). Then, Gertrude and Fred swap books. Then, Action 2 repeats. Then, Action 1 repeats. Then, Dave and Gertrude swap books. Then, Dave and Gertrude discuss something (let's call it Action 40). Then, Dave and Gertrude swap books. Then, nothing happens for a while. Then, nothing happens for a while. Then, Gertrude and Alice swap books. Then, Alice and Dave swap books. Then, Dave and Alice swap books. Then, Alice and Gertrude swap books. Then, Action 12 repeats. Then, Bob and Dave swap books. Then, Dave and Bob swap books. Then, Action 13 repeats. Then, Gertrude and Dave swap books. Then, Action 20 repeats. Then, Gertrude and Dave swap books. Then, Eve and Fred swap books (let's call it Action 41). Then, Fred and Alice swap books. Then, Action 27 repeats. Then, Fred and Eve swap books (let's call it Action 42). Then, Claire and Fred swap books (let's call it Action 43). Then, Fred and Claire swap books (let's call it Action 44). Then, Dave and Eve swap books. Then, Eve and Dave swap books (let's call it Action 45). Then, Bob and Eve swap books. Then, Bob and Eve discuss something (let's call it Action 46). Then, Bob and Eve swap books. Then, Alice and Dave swap books. Then, Dave and Alice swap books. Then, Action 14 repeats. Then, Dave and Alice swap books. Then, Gertrude and Eve swap books. Then, Eve and Bob swap books. Then, Bob and Eve swap books. Then, Eve and Gertrude swap books. Then, Action 14 repeats. Then, Alice and Dave discuss something. Then, Action 14 repeats. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Fred and Claire swap books. Then, Action 10 repeats. Then, Eve and Claire swap books. Then, Action 43 repeats. Then, Dave and Eve swap books. Then, Dave and Eve discuss something. Then, Dave and Eve swap books. Then, Bob and Eve swap books. Then, Eve and Claire swap books. Then, Claire and Eve swap books. Then, Eve and Bob swap books. Then, nothing happens for a while. Then, nothing happens for a while. Then, Bob and Dave swap books. Then, Bob and Dave discuss something (let's call it Action 47). Then, Bob and Dave swap books. Then, nothing happens for a while. Then, Alice and Bob swap books. Then, Action 13 repeats. Then, Action 43 repeats. Then, Claire and Fred discuss something (let's call it Action 48). Then, Claire and Fred swap books. Then, Action 5 repeats. Then, nothing happens for a while. Then, Action 41 repeats. Then, Fred and Claire swap books. Then, Action 43 repeats. Then, Action 42 repeats. Then, Action 26 repeats. Then, Alice and Claire swap books. Then, Claire and Alice swap books. Then, Action 27 repeats. Then, Alice and Claire swap books. Then, Alice and Claire discuss something (let's call it Action 49). Then, Alice and Claire swap books. Then, Action 5 repeats. Then, Dave and Claire swap books. Then, Claire and Dave swap books. Then, Action 31 repeats. Then, Gertrude and Claire discuss something (let's call it Action 50). Then, Action 31 repeats. Then, nothing happens for a while. Then, Action 5 repeats. Then, Gertrude and Eve swap books. Then, Eve and Gertrude swap books. Then, Claire and Dave swap books. Then, Dave and Alice swap books. Then, Alice and Dave swap books. Then, Dave and Claire swap books. Then, Claire and Alice swap books. Then, Alice and Claire swap books. Then, nothing happens for a while. Then, Dave and Fred swap books. Then, Dave and Fred discuss something (let's call it Action 51). Then, Dave and Fred swap books. Then, Fred and Eve swap books. Then, Fred and Eve discuss something (let's call it Action 52). Then, Fred and Eve swap books. Then, Dave and Gertrude swap books. Then, Dave and Gertrude discuss something. Then, Dave and Gertrude swap books. Then, Alice and Eve swap books (let's call it Action 53). Then, nothing happens for a while. Then, Alice and Fred swap books. Then, Fred and Gertrude swap books. Then, Gertrude and Fred swap books. Then, Fred and Alice swap books. Then, Claire and Dave swap books. Then, Dave and Claire swap books. Then, Alice and Dave swap books. Then, Dave and Alice swap books. Then, nothing happens for a while. Then, Eve and Fred swap books. Then, Action 44 repeats. Then, Claire and Fred swap books. Then, Action 42 repeats. Then, Action 5 repeats. Then, Fred and Eve swap books. Then, Action 35 repeats. Then, Gertrude and Eve swap books. Then, Eve and Fred swap books. Then, Fred and Gertrude swap books. Then, Fred and Gertrude discuss something (let's call it Action 54). Then, Action 9 repeats. Then, Action 25 repeats. Then, Dave and Fred discuss something. Then, Dave and Fred swap books. Then, Alice and Claire swap books. Then, Claire and Dave swap books. Then, Dave and Claire swap books. Then, Claire and Alice swap books. Then, Bob and Gertrude swap books. Then, Bob and Gertrude discuss something (let's call it Action 55). Then, Bob and Gertrude swap books. Then, Dave and Claire swap books. Then, Claire and Dave swap books. Then, Gertrude and Eve swap books. Then, Action 35 repeats. Then, Dave and Eve swap books. Then, Dave and Eve discuss something. Then, Dave and Eve swap books. Then, Eve and Gertrude swap books. Then, Eve and Gertrude discuss something (let's call it Action 56). Then, Eve and Gertrude swap books. Then, Action 5 repeats. Then, nothing happens for a while. Then, Alice and Dave swap books. Then, Alice and Dave discuss something. Then, Alice and Dave swap books. Then, Action 14 repeats. Then, Alice and Dave discuss something. Then, Alice and Dave swap books. Then, Claire and Bob swap books (let's call it Action 57). Then, Bob and Eve swap books. Then, Action 36 repeats. Then, Bob and Claire swap books (let's call it Action 58). Then, Action 35 repeats. Then, Gertrude and Eve swap books. Then, nothing happens for a while. Then, Fred and Dave swap books. Then, Action 21 repeats. Then, Eve and Dave swap books. Then, Action 25 repeats. Then, nothing happens for a while. Then, nothing happens for a while. Then, Action 37 repeats. Then, Eve and Bob swap books. Then, Action 5 repeats. Then, Dave and Gertrude swap books. Then, Claire and Bob swap books. Then, Bob and Claire swap books. Then, Alice and Bob swap books. Then, Bob and Fred swap books. Then, Action 6 repeats. Then, Bob and Alice swap books. Then, Action 37 repeats. Then, Bob and Eve discuss something. Then, Bob and Eve swap books. Then, nothing happens for a while. Then, nothing happens for a while. Then, Claire and Dave swap books. Then, Dave and Alice swap books. Then, Action 14 repeats. Then, Action 38 repeats. Then, Claire and Gertrude swap books. Then, Claire and Gertrude discuss something (let's call it Action 59). Then, Claire and Gertrude swap books. Then, Action 28 repeats. Then, Fred and Dave discuss something (let's call it Action 60). Then, Action 28 repeats. Then, nothing happens for a while. Then, Alice and Bob swap books. Then, Action 7 repeats. Then, Action 6 repeats. Then, Action 13 repeats. Then, nothing happens for a while. Then, Claire and Fred swap books. Then, Claire and Fred discuss something. Then, Claire and Fred swap books. Then, Dave and Claire swap books. Then, Dave and Claire discuss something (let's call it Action 61). Then, Dave and Claire swap books. Then, Dave and Gertrude swap books. Then, Action 8 repeats. Then, Action 9 repeats. Then, Gertrude and Dave swap books. Then, nothing happens for a while. Then, Eve and Gertrude swap books. Then, Gertrude and Alice swap books. Then, Alice and Gertrude swap books. Then, Gertrude and Eve swap books. Then, Bob and Gertrude swap books. Then, Action 55 repeats. Then, Bob and Gertrude swap books. Then, Claire and Alice swap books. Then, Action 29 repeats. Then, Dave and Alice swap books. Then, Alice and Fred swap books. Then, Action 26 repeats. Then, Action 14 repeats. Then, Action 18 repeats. Then, Bob and Dave swap books. Then, Bob and Alice swap books. Then, Alice and Gertrude swap books. Then, Action 24 repeats. Then, Alice and Bob swap books. Then, Gertrude and Dave swap books. Then, Action 18 repeats. Then, Bob and Dave swap books. Then, Action 1 repeats. Then, Bob and Eve swap books. Then, Bob and Eve discuss something. Then, Bob and Eve swap books. Then, Bob and Alice swap books. Then, Action 12 repeats. Then, Alice and Dave swap books. Then, Alice and Dave discuss something. Then, Alice and Dave swap books. Then, Action 5 repeats. Then, Claire and Dave swap books. Then, Dave and Alice swap books. Then, Action 14 repeats. Then, Dave and Claire swap books. Then, Fred and Dave swap books. Then, Dave and Fred swap books. Then, Gertrude and Claire swap books. Then, Alice and Claire swap books. Finally, Dave and Bob swap books. At the end of the semester, Bob has\\nOptions:\\n(A) The Great Gatsby\\n(B) Catch-22\\n(C) The Fellowship of the Ring\\n(D) Ulysses\\n(E) Frankenstein\\n(F) The Pearl\\n(G) The Odyssey\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}