{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"67f22746-ad78-599b-8ab4-6674afcf65c8","task_key":"bbeh~5fshuffled~5fobjects--full--67f22746-ad78-599b-8ab4-6674afcf65c8","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are dancers at a square dance.…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are dancers at a square dance. At the start of a song, they each have a partner: Alice is dancing with Karl, Bob is dancing with Ophelia, Claire is dancing with Helga, Dave is dancing with Izzi, Eve is dancing with Melissa, Fred is dancing with Sam, and Gertrude is dancing with Patrick.\\nThroughout the song, the dancers often trade partners. First, Claire and Bob switch partners. Then, Fred and Alice switch partners (let's call it Action 1). Then, Alice and Fred switch partners (let's call it Action 2). Then, Claire and Bob switch partners (let's call it Action 3). Then, Bob and Claire switch partners (let's call it Action 4). Then, Bob and Eve switch partners (let's call it Action 5). Then, Bob and Eve discuss something (let's call it Action 6). Then, Bob and Eve switch partners. Then, nothing happens for a while (let's call it Action 7). Then, nothing happens for a while. Then, Dave and Claire switch partners (let's call it Action 8). Then, Claire and Alice switch partners (let's call it Action 9). Then, Alice and Claire switch partners (let's call it Action 10). Then, Claire and Dave switch partners (let's call it Action 11). Then, Bob and Eve switch partners. Then, Eve and Bob switch partners (let's call it Action 12). Then, Action 10 repeats. Then, Alice and Claire discuss something (let's call it Action 13). Then, Alice and Claire switch partners. Then, Bob and Gertrude switch partners (let's call it Action 14). Then, Gertrude and Dave switch partners (let's call it Action 15). Then, Dave and Gertrude switch partners (let's call it Action 16). Then, Gertrude and Bob switch partners (let's call it Action 17). Then, Alice and Gertrude switch partners (let's call it Action 18). Then, Gertrude and Eve switch partners (let's call it Action 19). Then, Eve and Gertrude switch partners (let's call it Action 20). Then, Gertrude and Alice switch partners (let's call it Action 21). Then, nothing happens for a while. Then, nothing happens for a while. Then, Gertrude and Claire switch partners (let's call it Action 22). Then, Claire and Gertrude switch partners (let's call it Action 23). Then, Action 7 repeats. Then, Bob and Eve switch partners. Then, Action 6 repeats. Then, Bob and Eve switch partners. Then, Fred and Dave switch partners (let's call it Action 24). Then, Dave and Alice switch partners (let's call it Action 25). Then, Alice and Dave switch partners (let's call it Action 26). Then, Dave and Fred switch partners (let's call it Action 27). Then, nothing happens for a while. Then, Dave and Alice switch partners. Then, Alice and Claire switch partners. Then, Claire and Alice switch partners. Then, Action 26 repeats. Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Gertrude switch partners. Then, Alice and Gertrude discuss something (let's call it Action 28). Then, Alice and Gertrude switch partners. Then, Gertrude and Eve switch partners. Then, Eve and Fred switch partners (let's call it Action 29). Then, Fred and Eve switch partners (let's call it Action 30). Then, Action 20 repeats. Then, Action 29 repeats. Then, Fred and Bob switch partners (let's call it Action 31). Then, Bob and Fred switch partners (let's call it Action 32). Then, Fred and Eve switch partners. Then, Dave and Bob switch partners (let's call it Action 33). Then, Bob and Dave switch partners (let's call it Action 34). Then, Action 7 repeats. Then, Bob and Claire switch partners. Then, Claire and Eve switch partners (let's call it Action 35). Then, Eve and Claire switch partners (let's call it Action 36). Then, Claire and Bob switch partners. Then, Alice and Eve switch partners (let's call it Action 37). Then, Fred and Eve switch partners. Then, Action 29 repeats. Then, Dave and Fred switch partners. Then, Fred and Gertrude switch partners (let's call it Action 38). Then, Gertrude and Fred switch partners (let's call it Action 39). Then, Fred and Dave switch partners. Then, nothing happens for a while. Then, Claire and Fred switch partners (let's call it Action 40). Then, Claire and Fred discuss something (let's call it Action 41). Then, Action 40 repeats. Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Claire switch partners. Then, Claire and Fred switch partners. Then, Fred and Claire switch partners (let's call it Action 42). Then, Claire and Dave switch partners. Then, Dave and Alice switch partners. Then, Action 26 repeats. Then, Gertrude and Eve switch partners. Then, Eve and Dave switch partners (let's call it Action 43). Then, Dave and Eve switch partners (let's call it Action 44). Then, Eve and Gertrude switch partners. Then, Claire and Dave switch partners. Then, Dave and Claire switch partners. Then, Eve and Claire switch partners. Then, Action 35 repeats. Then, nothing happens for a while. Then, Action 36 repeats. Then, Eve and Claire discuss something (let's call it Action 45). Then, Action 36 repeats. Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Fred switch partners. Then, Fred and Dave switch partners. Then, Dave and Eve switch partners. Then, Dave and Eve discuss something (let's call it Action 46). Then, Action 44 repeats. Then, Eve and Fred switch partners. Then, Fred and Bob switch partners. Then, Action 32 repeats. Then, Action 30 repeats. Then, Gertrude and Alice switch partners. Then, Gertrude and Alice discuss something (let's call it Action 47). Then, Gertrude and Alice switch partners. Then, Gertrude and Fred switch partners. Then, Action 38 repeats. Then, Action 26 repeats. Then, Alice and Dave discuss something (let's call it Action 48). Then, Action 26 repeats. Then, Alice and Dave switch partners. Then, Alice and Dave discuss something. Then, Alice and Dave switch partners. Then, Gertrude and Bob switch partners. Then, Bob and Gertrude switch partners. Then, Alice and Dave switch partners. Then, Dave and Alice switch partners. Then, Bob and Eve switch partners. Then, Eve and Alice switch partners (let's call it Action 49). Then, Alice and Eve switch partners. Then, Eve and Bob switch partners. Then, Action 43 repeats. Then, Dave and Alice switch partners. Then, Alice and Dave switch partners. Then, Action 44 repeats. Then, nothing happens for a while. Then, Action 7 repeats. Then, Dave and Fred switch partners. Then, Fred and Alice switch partners. Then, Alice and Fred switch partners. Then, Fred and Dave switch partners. Then, Action 31 repeats. Then, Bob and Dave switch partners. Then, Dave and Claire switch partners. Then, Action 11 repeats. Then, Action 33 repeats. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Action 11 repeats. Then, Action 8 repeats. Then, Alice and Dave switch partners. Then, Dave and Fred switch partners. Then, Fred and Dave switch partners. Then, Dave and Alice switch partners. Then, nothing happens for a while. Then, Alice and Eve switch partners. Then, Alice and Eve discuss something (let's call it Action 50). Then, Action 37 repeats. Then, Alice and Bob switch partners (let's call it Action 51). Then, Action 34 repeats. Then, Action 33 repeats. Then, Bob and Alice switch partners (let's call it Action 52). Then, Dave and Alice switch partners. Then, Action 10 repeats. Then, Claire and Alice switch partners. Then, Alice and Dave switch partners. Then, nothing happens for a while. Then, Alice and Claire switch partners. Then, Action 9 repeats. Then, Action 2 repeats. Then, Alice and Fred discuss something (let's call it Action 53). Then, Alice and Fred switch partners. Then, Eve and Bob switch partners. Then, Action 34 repeats. Then, Dave and Bob switch partners. Then, Bob and Eve switch partners. Then, Fred and Alice switch partners. Then, Fred and Alice discuss something (let's call it Action 54). Then, Fred and Alice switch partners. Then, Gertrude and Eve switch partners. Then, Eve and Gertrude switch partners. Then, Fred and Gertrude switch partners. Then, Fred and Gertrude discuss something (let's call it Action 55). Then, Fred and Gertrude switch partners. Then, nothing happens for a while. Then, Gertrude and Fred switch partners. Then, Fred and Gertrude switch partners. Then, Eve and Claire switch partners. Then, Claire and Eve switch partners. Then, Dave and Bob switch partners. Then, Fred and Eve switch partners. Then, Fred and Eve discuss something (let's call it Action 56). Then, Fred and Eve switch partners. Then, Alice and Gertrude switch partners. Then, Gertrude and Alice switch partners. Then, nothing happens for a while. Then, Action 40 repeats. Then, Action 42 repeats. Then, Fred and Claire switch partners. Then, Action 35 repeats. Then, Action 36 repeats. Then, Claire and Fred switch partners. Then, Dave and Fred switch partners. Then, Fred and Bob switch partners. Then, Bob and Fred switch partners. Then, Action 24 repeats. Then, Action 7 repeats. Then, Action 7 repeats. Then, nothing happens for a while. Then, Claire and Dave switch partners. Then, Action 33 repeats. Then, Bob and Dave switch partners. Then, Dave and Claire switch partners. Then, Fred and Alice switch partners. Then, Action 2 repeats. Then, Action 7 repeats. Then, Alice and Fred switch partners. Then, Fred and Claire switch partners. Then, Claire and Fred switch partners. Then, Fred and Alice switch partners. Then, Eve and Gertrude switch partners. Then, Action 22 repeats. Then, Claire and Gertrude switch partners. Then, Gertrude and Eve switch partners. Then, Fred and Alice switch partners. Then, Fred and Alice discuss something. Then, Fred and Alice switch partners. Then, Bob and Gertrude switch partners. Then, Gertrude and Bob switch partners. Then, Eve and Claire switch partners. Then, Claire and Bob switch partners. Then, Bob and Claire switch partners. Then, Action 35 repeats. Then, Claire and Alice switch partners. Then, Alice and Claire switch partners. Then, nothing happens for a while. Then, Alice and Bob switch partners. Then, Alice and Bob discuss something (let's call it Action 57). Then, Action 51 repeats. Then, Alice and Fred switch partners. Then, Fred and Gertrude switch partners. Then, Gertrude and Fred switch partners. Then, Fred and Alice switch partners. Then, Bob and Claire switch partners. Then, Claire and Gertrude switch partners. Then, Gertrude and Claire switch partners. Then, Claire and Bob switch partners. Then, nothing happens for a while. Then, Action 7 repeats. Then, Action 12 repeats. Then, Eve and Bob discuss something (let's call it Action 58). Then, Eve and Bob switch partners. Then, Gertrude and Dave switch partners. Then, Dave and Bob switch partners. Then, Bob and Dave switch partners. Then, Dave and Gertrude switch partners. Then, Gertrude and Bob switch partners. Then, Gertrude and Bob discuss something (let's call it Action 59). Then, Gertrude and Bob switch partners. Then, Alice and Fred switch partners. Then, Alice and Fred discuss something. Then, Alice and Fred switch partners. Then, nothing happens for a while. Then, Action 40 repeats. Then, Action 41 repeats. Then, Claire and Fred switch partners. Then, Alice and Eve switch partners. Then, Alice and Eve discuss something. Then, Action 37 repeats. Then, Dave and Bob switch partners. Then, Dave and Bob discuss something (let's call it Action 60). Then, Dave and Bob switch partners. Then, Claire and Fred switch partners. Then, Fred and Alice switch partners. Then, Alice and Fred switch partners. Then, Action 42 repeats. Then, Action 15 repeats. Then, Dave and Claire switch partners. Then, Claire and Dave switch partners. Then, Dave and Gertrude switch partners. Then, Action 7 repeats. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Eve and Dave switch partners. Then, Action 44 repeats. Then, nothing happens for a while. Then, Dave and Fred switch partners. Then, Fred and Eve switch partners. Then, Eve and Fred switch partners. Then, Action 24 repeats. Then, Action 7 repeats. Then, Gertrude and Fred switch partners. Then, Gertrude and Fred discuss something (let's call it Action 61). Then, Gertrude and Fred switch partners. Then, Alice and Claire switch partners. Then, Claire and Fred switch partners. Then, Fred and Claire switch partners. Then, Action 9 repeats. Then, Eve and Gertrude switch partners. Then, Eve and Gertrude discuss something (let's call it Action 62). Then, Eve and Gertrude switch partners. Then, Claire and Dave switch partners. Then, Claire and Dave discuss something (let's call it Action 63). Then, Action 11 repeats. Then, Bob and Gertrude switch partners. Then, Gertrude and Alice switch partners. Then, Alice and Gertrude switch partners. Then, Gertrude and Bob switch partners. Then, Bob and Eve switch partners. Then, Eve and Dave switch partners. Then, Dave and Eve switch partners. Then, Action 12 repeats. Then, Eve and Bob switch partners. Then, Action 5 repeats. Then, Claire and Gertrude switch partners. Then, Bob and Eve switch partners. Finally, Claire and Eve switch partners. At the end of the dance, Fred is dancing with\\nOptions:\\n(A) Karl\\n(B) Ophelia\\n(C) Helga\\n(D) Izzi\\n(E) Melissa\\n(F) Sam\\n(G) Patrick\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}