{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"97ccc7f7-7f0a-5886-b352-e310f9484cbe","task_key":"bbeh~5fshuffled~5fobjects--full--97ccc7f7-7f0a-5886-b352-e310f9484cbe","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are dancers at a square dance.…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are dancers at a square dance. At the start of a song, they each have a partner: Alice is dancing with Melissa, Bob is dancing with Jamie, Claire is dancing with Sam, Dave is dancing with Izzi, Eve is dancing with Karl, Fred is dancing with Ophelia, and Gertrude is dancing with Helga.\\nThroughout the song, the dancers often trade partners. First, Dave and Gertrude switch partners. Then, Gertrude and Claire switch partners (let's call it Action 1). Then, Claire and Eve switch partners (let's call it Action 2). Then, Eve and Claire switch partners (let's call it Action 3). Then, Claire and Gertrude switch partners (let's call it Action 4). Then, Bob and Eve switch partners (let's call it Action 5). Then, Bob and Eve discuss something (let's call it Action 6). Then, Bob and Eve switch partners. Then, Fred and Eve switch partners (let's call it Action 7). Then, Eve and Fred switch partners (let's call it Action 8). Then, Gertrude and Bob switch partners (let's call it Action 9). Then, Gertrude and Bob discuss something (let's call it Action 10). Then, Action 9 repeats. Then, Dave and Bob switch partners (let's call it Action 11). Then, Bob and Dave switch partners (let's call it Action 12). Then, Claire and Bob switch partners (let's call it Action 13). Then, Claire and Bob discuss something (let's call it Action 14). Then, Claire and Bob switch partners. Then, Bob and Claire switch partners (let's call it Action 15). Then, Action 2 repeats. Then, Eve and Claire switch partners. Then, Claire and Bob switch partners. Then, Bob and Dave switch partners. Then, Bob and Dave discuss something (let's call it Action 16). Then, Action 12 repeats. Then, nothing happens for a while (let's call it Action 17). Then, Dave and Gertrude switch partners (let's call it Action 18). Then, Dave and Gertrude discuss something (let's call it Action 19). Then, Action 18 repeats. Then, Action 17 repeats. Then, Claire and Bob switch partners. Then, Claire and Bob discuss something. Then, Claire and Bob switch partners. Then, nothing happens for a while. Then, Gertrude and Fred switch partners (let's call it Action 20). Then, Gertrude and Fred discuss something (let's call it Action 21). Then, Gertrude and Fred switch partners. Then, Alice and Claire switch partners (let's call it Action 22). Then, Alice and Claire discuss something (let's call it Action 23). Then, Alice and Claire switch partners. Then, Alice and Fred switch partners (let's call it Action 24). Then, Fred and Alice switch partners (let's call it Action 25). Then, Action 17 repeats. Then, Alice and Bob switch partners (let's call it Action 26). Then, Bob and Alice switch partners (let's call it Action 27). Then, nothing happens for a while. Then, Eve and Claire switch partners. Then, Claire and Fred switch partners (let's call it Action 28). Then, Fred and Claire switch partners (let's call it Action 29). Then, Claire and Eve switch partners. Then, nothing happens for a while. Then, Action 9 repeats. Then, Action 2 repeats. Then, Eve and Gertrude switch partners (let's call it Action 30). Then, Gertrude and Eve switch partners (let's call it Action 31). Then, Eve and Claire switch partners. Then, Bob and Dave switch partners. Then, Action 11 repeats. Then, Action 17 repeats. Then, nothing happens for a while. Then, Eve and Bob switch partners (let's call it Action 32). Then, Bob and Gertrude switch partners (let's call it Action 33). Then, Gertrude and Bob switch partners. Then, Bob and Eve switch partners. Then, Fred and Dave switch partners (let's call it Action 34). Then, Dave and Fred switch partners (let's call it Action 35). Then, nothing happens for a while. Then, Bob and Gertrude switch partners. Then, Gertrude and Claire switch partners. Then, Action 4 repeats. Then, Gertrude and Bob switch partners. Then, Action 15 repeats. Then, Action 13 repeats. Then, Gertrude and Fred switch partners. Then, Gertrude and Fred discuss something. Then, Action 20 repeats. Then, Bob and Fred switch partners (let's call it Action 36). Then, Fred and Eve switch partners. Then, Action 8 repeats. Then, Fred and Bob switch partners (let's call it Action 37). Then, Alice and Gertrude switch partners (let's call it Action 38). Then, Gertrude and Bob switch partners. Then, Action 33 repeats. Then, Gertrude and Alice switch partners (let's call it Action 39). Then, Gertrude and Alice switch partners. Then, Alice and Bob switch partners. Then, Action 27 repeats. Then, Action 38 repeats. Then, Fred and Alice switch partners. Then, Action 24 repeats. Then, Action 9 repeats. Then, Action 36 repeats. Then, Fred and Bob switch partners. Then, Bob and Gertrude switch partners. Then, Dave and Claire switch partners (let's call it Action 40). Then, Claire and Dave switch partners (let's call it Action 41). Then, Alice and Eve switch partners (let's call it Action 42). Then, Alice and Eve discuss something (let's call it Action 43). Then, Alice and Eve switch partners. Then, Gertrude and Fred switch partners. Then, Action 21 repeats. Then, Gertrude and Fred switch partners. Then, Action 17 repeats. Then, Dave and Gertrude switch partners. Then, Gertrude and Dave switch partners (let's call it Action 44). Then, Alice and Bob switch partners. Then, Bob and Eve switch partners. Then, Eve and Bob switch partners. Then, Action 27 repeats. Then, Eve and Alice switch partners (let's call it Action 45). Then, Eve and Alice discuss something (let's call it Action 46). Then, Eve and Alice switch partners. Then, Alice and Fred switch partners. Then, Alice and Fred discuss something (let's call it Action 47). Then, Alice and Fred switch partners. Then, Action 5 repeats. Then, Eve and Bob switch partners. Then, nothing happens for a while. Then, Dave and Eve switch partners (let's call it Action 48). Then, Fred and Alice switch partners. Then, Alice and Eve switch partners. Then, Eve and Alice switch partners. Then, Alice and Fred switch partners. Then, Dave and Fred switch partners. Then, Dave and Fred discuss something (let's call it Action 49). Then, Dave and Fred switch partners. Then, Dave and Eve switch partners. Then, Dave and Eve discuss something (let's call it Action 50). Then, Dave and Eve switch partners. Then, nothing happens for a while. Then, nothing happens for a while. Then, Action 3 repeats. Then, Eve and Claire discuss something (let's call it Action 51). Then, Eve and Claire switch partners. Then, Dave and Alice switch partners (let's call it Action 52). Then, Alice and Dave switch partners (let's call it Action 53). Then, nothing happens for a while. Then, Eve and Fred switch partners. Then, Eve and Fred discuss something (let's call it Action 54). Then, Eve and Fred switch partners. Then, nothing happens for a while. Then, Dave and Alice switch partners. Then, Alice and Eve switch partners. Then, Eve and Alice switch partners. Then, Action 53 repeats. Then, nothing happens for a while. Then, Eve and Gertrude switch partners. Then, Gertrude and Fred switch partners. Then, Fred and Gertrude switch partners (let's call it Action 55). Then, Gertrude and Eve switch partners. Then, Gertrude and Eve switch partners. Then, Eve and Alice switch partners. Then, Action 42 repeats. Then, Eve and Gertrude switch partners. Then, Eve and Alice switch partners. Then, Action 42 repeats. Then, Alice and Claire switch partners. Then, Claire and Alice switch partners (let's call it Action 56). Then, Action 17 repeats. Then, Dave and Eve switch partners. Then, Eve and Gertrude switch partners. Then, Gertrude and Eve switch partners. Then, Eve and Dave switch partners (let's call it Action 57). Then, Claire and Eve switch partners. Then, Eve and Gertrude switch partners. Then, Gertrude and Eve switch partners. Then, Action 3 repeats. Then, Alice and Fred switch partners. Then, Alice and Fred discuss something. Then, Alice and Fred switch partners. Then, Alice and Fred switch partners. Then, Alice and Fred discuss something. Then, Alice and Fred switch partners. Then, Bob and Dave switch partners. Then, Action 16 repeats. Then, Action 12 repeats. Then, Gertrude and Bob switch partners. Then, Bob and Gertrude switch partners. Then, Action 5 repeats. Then, Eve and Alice switch partners. Then, Alice and Eve switch partners. Then, Eve and Bob switch partners. Then, nothing happens for a while. Then, Action 17 repeats. Then, nothing happens for a while. Then, nothing happens for a while. Then, Action 40 repeats. Then, Claire and Dave switch partners. Then, Alice and Claire switch partners. Then, Claire and Alice switch partners. Then, Action 20 repeats. Then, Action 7 repeats. Then, Eve and Fred switch partners. Then, Action 55 repeats. Then, Dave and Alice switch partners. Then, Dave and Alice discuss something (let's call it Action 58). Then, Dave and Alice switch partners. Then, Claire and Dave switch partners. Then, Dave and Claire switch partners. Then, nothing happens for a while. Then, Fred and Bob switch partners. Then, Alice and Eve switch partners. Then, Eve and Alice switch partners. Then, Bob and Gertrude switch partners. Then, Bob and Gertrude discuss something (let's call it Action 59). Then, Bob and Gertrude switch partners. Then, Action 20 repeats. Then, Fred and Gertrude switch partners. Then, Bob and Fred switch partners. Then, Action 7 repeats. Then, Eve and Fred switch partners. Then, Action 37 repeats. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Gertrude and Alice switch partners. Then, Alice and Fred switch partners. Then, Fred and Alice switch partners. Then, Alice and Gertrude switch partners. Then, Action 39 repeats. Then, Alice and Gertrude switch partners. Then, Eve and Alice switch partners. Then, Alice and Eve switch partners. Then, nothing happens for a while. Then, nothing happens for a while. Then, Eve and Dave switch partners. Then, Eve and Dave discuss something (let's call it Action 60). Then, Eve and Dave switch partners. Then, Eve and Alice switch partners. Then, Alice and Claire switch partners. Then, Claire and Alice switch partners. Then, Alice and Eve switch partners. Then, Gertrude and Claire switch partners. Then, Claire and Gertrude switch partners. Then, Eve and Bob switch partners. Then, Eve and Bob discuss something (let's call it Action 61). Then, Action 32 repeats. Then, Gertrude and Dave switch partners. Then, Action 40 repeats. Then, Action 41 repeats. Then, Action 18 repeats. Then, Action 27 repeats. Then, Bob and Alice discuss something (let's call it Action 62). Then, Bob and Alice switch partners. Then, Action 34 repeats. Then, Fred and Dave discuss something (let's call it Action 63). Then, Fred and Dave switch partners. Then, Fred and Eve switch partners. Then, Eve and Dave switch partners. Then, Dave and Eve switch partners. Then, Eve and Fred switch partners. Then, Claire and Fred switch partners. Then, Fred and Claire switch partners. Then, Fred and Alice switch partners. Then, Fred and Alice discuss something (let's call it Action 64). Then, Fred and Alice switch partners. Then, Bob and Eve switch partners. Then, Eve and Bob switch partners. Then, nothing happens for a while. Then, nothing happens for a while. Then, Action 56 repeats. Then, Alice and Bob switch partners. Then, Bob and Alice switch partners. Then, Alice and Claire switch partners. Then, Action 22 repeats. Then, Alice and Claire discuss something. Then, Action 22 repeats. Then, Alice and Claire switch partners. Then, Bob and Dave switch partners. Finally, Eve and Alice switch partners. At the end of the dance, Fred is dancing with\\nOptions:\\n(A) Melissa\\n(B) Jamie\\n(C) Sam\\n(D) Izzi\\n(E) Karl\\n(F) Ophelia\\n(G) Helga\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}