{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3e9065b4-4969-5d1a-b3ec-f7809a626968","task_key":"bbeh~5fshuffled~5fobjects--full--3e9065b4-4969-5d1a-b3ec-f7809a626968","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are friends and avid readers…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are friends and avid readers who occasionally trade books. At the start of the semester, they each buy one new book: Alice gets The Great Gatsby, Bob gets Catch-22, Claire gets The Fellowship of the Ring, Dave gets Ulysses, Eve gets Frankenstein, Fred gets The Pearl, and Gertrude gets The Odyssey.\\nAs the semester proceeds, they start trading around the new books. First, Fred and Alice swap books. Then, nothing happens for a while (let's call it Action 1). Then, Dave and Alice swap books (let's call it Action 2). Then, Alice and Dave swap books (let's call it Action 3). Then, Claire and Bob swap books (let's call it Action 4). Then, Bob and Claire swap books (let's call it Action 5). Then, Dave and Claire swap books (let's call it Action 6). Then, Claire and Eve swap books (let's call it Action 7). Then, Eve and Claire swap books (let's call it Action 8). Then, Claire and Dave swap books (let's call it Action 9). Then, Alice and Fred swap books (let's call it Action 10). Then, Fred and Alice swap books (let's call it Action 11). Then, Action 4 repeats. Then, Bob and Eve swap books (let's call it Action 12). Then, Eve and Bob swap books (let's call it Action 13). Then, Bob and Claire swap books. Then, Dave and Claire swap books. Then, Claire and Alice swap books (let's call it Action 14). Then, Alice and Claire swap books (let's call it Action 15). Then, Claire and Dave swap books. Then, Eve and Alice swap books (let's call it Action 16). Then, Alice and Eve swap books (let's call it Action 17). Then, nothing happens for a while. Then, nothing happens for a while. Then, Bob and Fred swap books (let's call it Action 18). Then, Fred and Claire swap books (let's call it Action 19). Then, Claire and Fred swap books (let's call it Action 20). Then, Fred and Bob swap books (let's call it Action 21). Then, Alice and Dave swap books. Then, Dave and Alice swap books. Then, Action 17 repeats. Then, Action 16 repeats. Then, Alice and Gertrude swap books (let's call it Action 22). Then, Gertrude and Eve swap books (let's call it Action 23). Then, Eve and Gertrude swap books (let's call it Action 24). Then, Gertrude and Alice swap books (let's call it Action 25). Then, Action 22 repeats. Then, Gertrude and Dave swap books (let's call it Action 26). Then, Dave and Gertrude swap books (let's call it Action 27). Then, Action 25 repeats. Then, Fred and Bob swap books. Then, Bob and Fred swap books. Then, Claire and Gertrude swap books (let's call it Action 28). Then, Gertrude and Claire swap books (let's call it Action 29). Then, nothing happens for a while. Then, Action 7 repeats. Then, Eve and Gertrude swap books. Then, Gertrude and Eve swap books. Then, Action 8 repeats. Then, Alice and Bob swap books (let's call it Action 30). Then, Bob and Alice swap books (let's call it Action 31). Then, nothing happens for a while. Then, Action 1 repeats. Then, nothing happens for a while. Then, Action 10 repeats. Then, Alice and Fred discuss something (let's call it Action 32). Then, Alice and Fred swap books. Then, Fred and Eve swap books (let's call it Action 33). Then, Eve and Claire swap books. Then, Action 7 repeats. Then, Eve and Fred swap books (let's call it Action 34). Then, Dave and Fred swap books (let's call it Action 35). Then, Dave and Fred discuss something (let's call it Action 36). Then, Dave and Fred swap books. Then, Dave and Bob swap books (let's call it Action 37). Then, Bob and Dave swap books (let's call it Action 38). Then, Fred and Eve swap books. Then, Eve and Fred swap books. Then, Gertrude and Eve swap books. Then, Eve and Bob swap books. Then, Bob and Eve swap books. Then, Action 24 repeats. Then, Bob and Gertrude swap books (let's call it Action 39). Then, Gertrude and Eve swap books. Then, Eve and Gertrude swap books. Then, Gertrude and Bob swap books (let's call it Action 40). Then, Eve and Alice swap books. Then, Eve and Alice discuss something (let's call it Action 41). Then, Eve and Alice swap books. Then, Fred and Alice swap books. Then, Alice and Fred swap books. Then, Eve and Claire swap books. Then, Action 28 repeats. Then, Gertrude and Claire swap books. Then, Claire and Eve swap books. Then, Action 1 repeats. Then, Alice and Claire swap books. Then, Claire and Eve swap books. Then, Eve and Claire swap books. Then, Claire and Alice swap books. Then, Action 14 repeats. Then, Alice and Claire swap books. Then, Action 23 repeats. Then, nothing happens for a while. Then, Action 28 repeats. Then, Gertrude and Claire swap books. Then, Action 7 repeats. Then, Claire and Eve discuss something (let's call it Action 42). Then, Action 7 repeats. Then, Dave and Alice swap books. Then, Action 15 repeats. Then, Claire and Alice swap books. Then, Alice and Dave swap books. Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Fred swap books. Then, Fred and Claire swap books. Then, Claire and Fred swap books. Then, Fred and Dave swap books (let's call it Action 43). Then, Fred and Gertrude swap books (let's call it Action 44). Then, Gertrude and Fred swap books (let's call it Action 45). Then, Bob and Eve swap books. Then, Bob and Eve discuss something (let's call it Action 46). Then, Bob and Eve swap books. Then, nothing happens for a while. Then, Fred and Eve swap books. Then, Fred and Eve discuss something (let's call it Action 47). Then, Action 33 repeats. Then, Action 6 repeats. Then, Dave and Claire discuss something (let's call it Action 48). Then, Dave and Claire swap books. Then, Bob and Fred swap books. Then, Bob and Fred discuss something (let's call it Action 49). Then, Action 18 repeats. Then, Bob and Eve swap books. Then, Action 46 repeats. Then, Bob and Eve swap books. Then, Bob and Eve swap books. Then, Eve and Bob swap books. Then, Alice and Fred swap books. Then, Fred and Alice swap books. Then, Fred and Alice swap books. Then, Action 10 repeats. Then, nothing happens for a while. Then, Eve and Dave swap books (let's call it Action 50). Then, Action 35 repeats. Then, Action 43 repeats. Then, Dave and Eve swap books (let's call it Action 51). Then, Fred and Dave swap books. Then, Action 35 repeats. Then, nothing happens for a while. Then, nothing happens for a while. Then, Gertrude and Alice swap books. Then, Alice and Gertrude swap books. Then, Gertrude and Alice swap books. Then, Alice and Claire swap books. Then, Claire and Alice swap books. Then, Action 22 repeats. Then, Bob and Gertrude swap books. Then, Action 25 repeats. Then, Alice and Gertrude swap books. Then, Action 40 repeats. Then, Dave and Alice swap books. Then, Action 3 repeats. Then, Eve and Gertrude swap books. Then, Gertrude and Fred swap books. Then, Fred and Gertrude swap books. Then, Action 23 repeats. Then, nothing happens for a while. Then, Eve and Dave swap books. Then, Dave and Eve swap books. Then, Gertrude and Dave swap books. Then, Dave and Bob swap books. Then, Bob and Dave swap books. Then, Dave and Gertrude swap books. Then, Fred and Dave swap books. Then, Action 35 repeats. Then, Dave and Bob swap books. Then, Dave and Bob discuss something (let's call it Action 52). Then, Dave and Bob swap books. Then, Action 7 repeats. Then, Eve and Gertrude swap books. Then, Gertrude and Eve swap books. Then, Eve and Claire swap books. Then, Eve and Claire swap books. Then, Action 20 repeats. Then, Fred and Claire swap books. Then, Claire and Eve swap books. Then, nothing happens for a while. Then, Gertrude and Claire swap books. Then, Claire and Bob swap books. Then, Bob and Claire swap books. Then, Claire and Gertrude swap books. Then, nothing happens for a while. Then, Action 18 repeats. Then, Fred and Eve swap books. Then, Action 34 repeats. Then, Fred and Bob swap books. Then, nothing happens for a while. Then, Claire and Bob swap books. Then, Claire and Bob discuss something (let's call it Action 53). Then, Claire and Bob swap books. Then, nothing happens for a while. Then, nothing happens for a while. Then, Fred and Bob swap books. Then, Bob and Fred swap books. Then, Gertrude and Eve swap books. Then, Gertrude and Eve discuss something (let's call it Action 54). Then, Gertrude and Eve swap books. Then, Claire and Bob swap books. Then, Claire and Bob discuss something. Then, Action 4 repeats. Then, nothing happens for a while. Then, Dave and Alice swap books. Then, Alice and Dave swap books. Then, Bob and Claire swap books. Then, Claire and Gertrude swap books. Then, Gertrude and Claire swap books. Then, Claire and Bob swap books. Then, Alice and Eve swap books. Then, Action 19 repeats. Then, Fred and Claire discuss something (let's call it Action 55). Then, Fred and Claire swap books. Then, Action 44 repeats. Then, Action 45 repeats. Then, Claire and Bob swap books. Then, Bob and Claire swap books. Then, Fred and Dave swap books. Then, Fred and Dave discuss something (let's call it Action 56). Then, Fred and Dave swap books. Then, Action 29 repeats. Then, Gertrude and Claire discuss something (let's call it Action 57). Then, Gertrude and Claire swap books. Then, Action 1 repeats. Then, Dave and Claire swap books. Then, Action 48 repeats. Then, Dave and Claire swap books. Then, Alice and Dave swap books. Then, Action 2 repeats. Then, Bob and Claire swap books. Then, Action 9 repeats. Then, Dave and Claire swap books. Then, Action 4 repeats. Then, Dave and Fred swap books. Then, Dave and Fred discuss something. Then, Action 35 repeats. Then, Alice and Gertrude swap books. Then, Action 29 repeats. Then, Claire and Gertrude swap books. Then, Action 25 repeats. Then, Action 7 repeats. Then, Eve and Claire swap books. Then, nothing happens for a while. Then, Dave and Eve swap books. Then, Dave and Eve discuss something (let's call it Action 58). Then, Action 51 repeats. Then, Dave and Claire swap books. Then, Claire and Dave swap books. Then, nothing happens for a while. Then, Bob and Fred swap books. Then, Bob and Fred discuss something. Then, Bob and Fred swap books. Then, Action 1 repeats. Then, Action 1 repeats. Then, Bob and Dave swap books. Then, Dave and Bob swap books. Then, Fred and Claire swap books. Then, Fred and Claire discuss something. Then, Action 19 repeats. Then, Action 18 repeats. Then, Fred and Bob swap books. Then, Action 9 repeats. Then, Claire and Dave discuss something (let's call it Action 59). Then, Claire and Dave swap books. Then, Dave and Fred swap books. Then, Fred and Gertrude swap books. Then, Action 45 repeats. Then, Fred and Dave swap books. Then, Alice and Eve swap books. Then, Eve and Alice swap books. Then, Action 8 repeats. Then, Claire and Fred swap books. Then, Fred and Claire swap books. Then, Claire and Eve swap books. Then, Dave and Claire swap books. Then, Dave and Claire discuss something. Then, Dave and Claire swap books. Then, nothing happens for a while. Then, Dave and Gertrude swap books. Then, Dave and Gertrude discuss something (let's call it Action 60). Then, Dave and Gertrude swap books. Then, nothing happens for a while. Then, Bob and Alice swap books. Then, Alice and Dave swap books. Then, Dave and Alice swap books. Then, Alice and Bob swap books. Then, nothing happens for a while. Then, Action 27 repeats. Then, nothing happens for a while. Then, Alice and Bob swap books. Then, Bob and Alice swap books. Then, Claire and Alice swap books. Then, Claire and Alice discuss something (let's call it Action 61). Then, Action 14 repeats. Then, Action 9 repeats. Then, Action 59 repeats. Then, Claire and Dave swap books. Then, Claire and Alice swap books. Then, Claire and Alice discuss something. Then, Claire and Alice swap books. Then, Claire and Eve swap books. Then, Action 16 repeats. Then, Alice and Eve swap books. Then, Eve and Claire swap books. Then, nothing happens for a while. Then, Gertrude and Eve swap books. Then, Eve and Claire swap books. Then, Claire and Eve swap books. Then, Action 24 repeats. Then, nothing happens for a while. Then, Action 1 repeats. Then, Dave and Gertrude swap books. Then, Action 60 repeats. Then, Dave and Gertrude swap books. Then, Claire and Alice swap books. Then, Alice and Bob swap books. Then, Action 31 repeats. Then, Alice and Claire swap books. Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Bob swap books. Then, Alice and Bob discuss something (let's call it Action 62). Then, Alice and Bob swap books. Then, Bob and Eve swap books. Then, Eve and Bob swap books. Then, Gertrude and Bob swap books. Then, Gertrude and Bob discuss something (let's call it Action 63). Then, Gertrude and Bob swap books. Then, Alice and Dave swap books. Then, Dave and Alice swap books. Then, Action 1 repeats. Then, Eve and Gertrude swap books. Then, Eve and Gertrude discuss something (let's call it Action 64). Then, Action 24 repeats. Then, Action 29 repeats. Then, Alice and Claire swap books. Finally, Dave and Bob swap books. At the end of the semester, Dave has\\nOptions:\\n(A) The Great Gatsby\\n(B) Catch-22\\n(C) The Fellowship of the Ring\\n(D) Ulysses\\n(E) Frankenstein\\n(F) The Pearl\\n(G) The Odyssey\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}