{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1c7c277e-9bba-5af0-8dc1-ba72a8a3d4d3","task_key":"bbeh~5fshuffled~5fobjects--full--1c7c277e-9bba-5af0-8dc1-ba72a8a3d4d3","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are friends and avid readers…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are friends and avid readers who occasionally trade books. At the start of the semester, they each buy one new book: Alice gets Hound of the Baskervilles, Bob gets Lolita, Claire gets The Great Gatsby, Dave gets The Odyssey, Eve gets Ulysses, Fred gets The Pearl, and Gertrude gets Frankenstein.\\nAs the semester proceeds, they start trading around the new books. First, Gertrude and Fred swap books. Then, Gertrude and Eve swap books (let's call it Action 1). Then, Gertrude and Eve discuss something (let's call it Action 2). Then, Gertrude and Eve swap books. Then, nothing happens for a while (let's call it Action 3). Then, Alice and Bob swap books (let's call it Action 4). Then, Bob and Alice swap books (let's call it Action 5). Then, Eve and Dave swap books (let's call it Action 6). Then, Dave and Fred swap books (let's call it Action 7). Then, Fred and Dave swap books (let's call it Action 8). Then, Dave and Eve swap books (let's call it Action 9). Then, Claire and Eve swap books (let's call it Action 10). Then, Eve and Claire swap books (let's call it Action 11). Then, Claire and Dave swap books (let's call it Action 12). Then, Dave and Claire swap books (let's call it Action 13). Then, Dave and Claire swap books. Then, Claire and Bob swap books (let's call it Action 14). Then, Bob and Claire swap books (let's call it Action 15). Then, Action 12 repeats. Then, nothing happens for a while. Then, Gertrude and Claire swap books (let's call it Action 16). Then, Claire and Gertrude swap books (let's call it Action 17). Then, nothing happens for a while. Then, Claire and Gertrude swap books. Then, Claire and Gertrude discuss something (let's call it Action 18). Then, Claire and Gertrude swap books. Then, Bob and Eve swap books (let's call it Action 19). Then, Bob and Eve discuss something (let's call it Action 20). Then, Action 19 repeats. Then, Gertrude and Claire swap books. Then, Gertrude and Claire discuss something (let's call it Action 21). Then, Gertrude and Claire swap books. Then, Gertrude and Claire swap books. Then, Claire and Gertrude swap books. Then, nothing happens for a while. Then, Action 19 repeats. Then, Bob and Eve discuss something. Then, Bob and Eve swap books. Then, Bob and Eve swap books. Then, Action 11 repeats. Then, Claire and Eve swap books. Then, Eve and Bob swap books (let's call it Action 22). Then, Dave and Bob swap books (let's call it Action 23). Then, Bob and Fred swap books (let's call it Action 24). Then, Fred and Bob swap books (let's call it Action 25). Then, Bob and Dave swap books (let's call it Action 26). Then, Fred and Claire swap books (let's call it Action 27). Then, Claire and Eve swap books. Then, Eve and Claire swap books. Then, Claire and Fred swap books (let's call it Action 28). Then, Dave and Alice swap books (let's call it Action 29). Then, Alice and Eve swap books (let's call it Action 30). Then, Eve and Alice swap books (let's call it Action 31). Then, Alice and Dave swap books (let's call it Action 32). Then, Gertrude and Fred swap books (let's call it Action 33). Then, Fred and Claire swap books. Then, Claire and Fred swap books. Then, Fred and Gertrude swap books (let's call it Action 34). Then, nothing happens for a while. Then, Fred and Claire swap books. Then, Fred and Claire discuss something (let's call it Action 35). Then, Fred and Claire swap books. Then, Claire and Fred swap books. Then, Claire and Fred discuss something (let's call it Action 36). Then, Claire and Fred swap books. Then, nothing happens for a while. Then, Claire and Eve swap books. Then, Eve and Alice swap books. Then, Alice and Eve swap books. Then, Eve and Claire swap books. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Fred and Bob swap books. Then, Bob and Fred swap books. Then, Bob and Fred swap books. Then, Bob and Fred discuss something (let's call it Action 37). Then, Action 24 repeats. Then, Dave and Eve swap books. Then, Eve and Dave swap books. Then, nothing happens for a while. Then, Dave and Gertrude swap books (let's call it Action 38). Then, Gertrude and Claire swap books. Then, Claire and Gertrude swap books. Then, nothing happens for a while. Then, Fred and Gertrude swap books. Then, Gertrude and Fred swap books. Then, Action 23 repeats. Then, Dave and Bob discuss something (let's call it Action 39). Then, Dave and Bob swap books. Then, Bob and Gertrude swap books (let's call it Action 40). Then, Gertrude and Bob swap books (let's call it Action 41). Then, nothing happens for a while. Then, nothing happens for a while. Then, Action 41 repeats. Then, Bob and Gertrude swap books. Then, Action 3 repeats. Then, Gertrude and Dave swap books (let's call it Action 42). Then, Dave and Fred swap books. Then, Fred and Dave swap books. Then, Dave and Gertrude swap books. Then, Action 4 repeats. Then, Bob and Alice swap books. Then, Claire and Bob swap books. Then, Claire and Bob discuss something (let's call it Action 43). Then, Claire and Bob swap books. Then, Eve and Claire swap books. Then, Claire and Eve swap books. Then, Eve and Bob swap books. Then, Eve and Bob discuss something (let's call it Action 44). Then, Eve and Bob swap books. Then, Eve and Claire swap books. Then, Action 10 repeats. Then, Dave and Fred swap books. Then, Dave and Fred discuss something (let's call it Action 45). Then, Dave and Fred swap books. Then, Action 16 repeats. Then, Claire and Bob swap books. Then, Bob and Claire swap books. Then, Claire and Gertrude swap books. Then, Fred and Claire swap books. Then, Action 35 repeats. Then, Fred and Claire swap books. Then, Action 3 repeats. Then, Alice and Dave swap books. Then, Dave and Alice swap books. Then, nothing happens for a while. Then, Action 30 repeats. Then, Alice and Eve discuss something (let's call it Action 46). Then, Action 30 repeats. Then, Alice and Dave swap books. Then, Dave and Alice swap books. Then, Dave and Claire swap books. Then, nothing happens for a while. Then, Dave and Eve swap books. Then, Eve and Alice swap books. Then, Alice and Eve swap books. Then, Eve and Dave swap books. Then, Claire and Alice swap books (let's call it Action 47). Then, Claire and Alice discuss something (let's call it Action 48). Then, Action 47 repeats. Then, nothing happens for a while. Then, Fred and Claire swap books. Then, Claire and Bob swap books. Then, Bob and Claire swap books. Then, Claire and Fred swap books. Then, Eve and Alice swap books. Then, Action 30 repeats. Then, Alice and Dave swap books. Then, Dave and Alice swap books. Then, Action 32 repeats. Then, Alice and Dave discuss something (let's call it Action 49). Then, Action 32 repeats. Then, nothing happens for a while. Then, Gertrude and Eve swap books. Then, Gertrude and Eve discuss something. Then, Gertrude and Eve swap books. Then, nothing happens for a while. Then, Dave and Eve swap books. Then, Eve and Gertrude swap books (let's call it Action 50). Then, Gertrude and Eve swap books. Then, Eve and Dave swap books. Then, Gertrude and Fred swap books. Then, Gertrude and Fred discuss something (let's call it Action 51). Then, Gertrude and Fred swap books. Then, Eve and Dave swap books. Then, Dave and Eve swap books. Then, Action 11 repeats. Then, Claire and Eve swap books. Then, Gertrude and Claire swap books. Then, Action 21 repeats. Then, Gertrude and Claire swap books. Then, nothing happens for a while. Then, Bob and Fred swap books. Then, Fred and Bob swap books. Then, Fred and Gertrude swap books. Then, Gertrude and Fred swap books. Then, nothing happens for a while. Then, Fred and Eve swap books (let's call it Action 52). Then, Eve and Alice swap books. Then, Alice and Eve swap books. Then, Eve and Fred swap books (let's call it Action 53). Then, Eve and Bob swap books. Then, Bob and Fred swap books. Then, Fred and Bob swap books. Then, Bob and Eve swap books. Then, Fred and Dave swap books. Then, Dave and Fred swap books. Then, Bob and Alice swap books. Then, Alice and Claire swap books (let's call it Action 54). Then, Claire and Alice swap books. Then, Action 4 repeats. Then, Eve and Bob swap books. Then, Action 19 repeats. Then, Fred and Gertrude swap books. Then, Gertrude and Bob swap books. Then, Action 40 repeats. Then, Action 33 repeats. Then, nothing happens for a while. Then, Fred and Claire swap books. Then, Action 47 repeats. Then, Alice and Claire swap books. Then, Claire and Fred swap books. Then, Alice and Gertrude swap books (let's call it Action 55). Then, Gertrude and Alice swap books (let's call it Action 56). Then, Dave and Eve swap books. Then, Dave and Eve discuss something (let's call it Action 57). Then, Dave and Eve swap books. Then, Action 23 repeats. Then, Bob and Fred swap books. Then, Action 25 repeats. Then, Bob and Dave swap books. Then, Fred and Gertrude swap books. Then, Action 33 repeats. Then, Alice and Bob swap books. Then, Fred and Bob swap books. Then, Bob and Fred swap books. Then, Dave and Eve swap books. Then, Eve and Dave swap books. Then, Eve and Alice swap books. Then, Alice and Gertrude swap books. Then, Gertrude and Alice swap books. Then, Alice and Eve swap books. Then, Bob and Gertrude swap books. Then, Gertrude and Alice swap books. Then, Alice and Gertrude swap books. Then, Gertrude and Bob swap books. Then, Bob and Eve swap books. Then, Bob and Eve discuss something. Then, Bob and Eve swap books. Then, nothing happens for a while. Then, nothing happens for a while. Then, Action 17 repeats. Then, Claire and Gertrude discuss something. Then, Claire and Gertrude swap books. Then, Alice and Gertrude swap books. Then, Gertrude and Alice swap books. Then, nothing happens for a while. Then, Gertrude and Fred swap books. Then, Fred and Gertrude swap books. Then, nothing happens for a while. Then, Dave and Gertrude swap books. Then, Dave and Gertrude discuss something (let's call it Action 58). Then, Dave and Gertrude swap books. Then, Action 9 repeats. Then, Eve and Dave swap books. Then, nothing happens for a while. Then, Action 42 repeats. Then, Gertrude and Dave discuss something (let's call it Action 59). Then, Gertrude and Dave swap books. Then, Claire and Eve swap books. Then, Eve and Alice swap books. Then, Action 30 repeats. Then, Eve and Claire swap books. Then, Fred and Bob swap books. Then, Fred and Bob discuss something (let's call it Action 60). Then, Fred and Bob swap books. Then, Action 31 repeats. Then, Eve and Alice discuss something (let's call it Action 61). Then, Eve and Alice swap books. Then, Alice and Bob swap books. Then, Alice and Bob discuss something (let's call it Action 62). Then, Alice and Bob swap books. Then, Alice and Claire swap books. Then, Claire and Dave swap books. Then, Dave and Claire swap books. Then, Claire and Alice swap books. Then, Gertrude and Dave swap books. Then, Gertrude and Dave discuss something. Then, Gertrude and Dave swap books. Then, Action 15 repeats. Then, Claire and Bob swap books. Then, Gertrude and Fred swap books. Then, Fred and Gertrude swap books. Then, Action 4 repeats. Then, Action 62 repeats. Then, Alice and Bob swap books. Then, Bob and Gertrude swap books. Then, Gertrude and Eve swap books. Then, Eve and Gertrude swap books. Then, Gertrude and Bob swap books. Then, nothing happens for a while. Then, Dave and Fred swap books. Then, Action 45 repeats. Then, Action 7 repeats. Then, Action 19 repeats. Then, Action 50 repeats. Then, Gertrude and Eve swap books. Then, Action 22 repeats. Then, Gertrude and Fred swap books. Then, Gertrude and Fred discuss something. Then, Gertrude and Fred swap books. Then, Alice and Dave swap books. Then, Dave and Gertrude swap books. Then, Gertrude and Dave swap books. Then, Dave and Alice swap books. Then, Eve and Bob swap books. Then, Eve and Bob discuss something. Then, Eve and Bob swap books. Then, Dave and Alice swap books. Then, Alice and Dave swap books. Then, Dave and Claire swap books. Then, Action 47 repeats. Then, Alice and Claire swap books. Then, Claire and Dave swap books. Then, nothing happens for a while. Then, Gertrude and Claire swap books. Then, Action 47 repeats. Then, Action 54 repeats. Then, Claire and Gertrude swap books. Then, nothing happens for a while. Then, Action 11 repeats. Then, Alice and Gertrude swap books. Finally, Bob and Alice swap books. At the end of the semester, Claire has\\nOptions:\\n(A) Hound of the Baskervilles\\n(B) Lolita\\n(C) The Great Gatsby\\n(D) The Odyssey\\n(E) Ulysses\\n(F) The Pearl\\n(G) Frankenstein\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}