{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1201544b-4f36-54b7-abd9-2515f2bbe5d2","task_key":"bbeh~5fshuffled~5fobjects--full--1201544b-4f36-54b7-abd9-2515f2bbe5d2","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are dancers at a square dance.…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are dancers at a square dance. At the start of a song, they each have a partner: Alice is dancing with Izzi, Bob is dancing with Sam, Claire is dancing with Helga, Dave is dancing with Jamie, Eve is dancing with Rodrigo, Fred is dancing with Ophelia, and Gertrude is dancing with Melissa.\\nThroughout the song, the dancers often trade partners. First, Gertrude and Eve switch partners. Then, Eve and Fred switch partners (let's call it Action 1). Then, Eve and Fred discuss something (let's call it Action 2). Then, Eve and Fred switch partners. Then, Alice and Dave switch partners (let's call it Action 3). Then, Dave and Alice switch partners (let's call it Action 4). Then, Dave and Gertrude switch partners (let's call it Action 5). Then, Gertrude and Bob switch partners (let's call it Action 6). Then, Bob and Gertrude switch partners (let's call it Action 7). Then, Gertrude and Dave switch partners (let's call it Action 8). Then, nothing happens for a while (let's call it Action 9). Then, Dave and Eve switch partners (let's call it Action 10). Then, Eve and Fred switch partners. Then, Fred and Eve switch partners (let's call it Action 11). Then, Eve and Dave switch partners (let's call it Action 12). Then, Fred and Alice switch partners (let's call it Action 13). Then, Alice and Gertrude switch partners (let's call it Action 14). Then, Gertrude and Alice switch partners (let's call it Action 15). Then, Alice and Fred switch partners (let's call it Action 16). Then, Fred and Gertrude switch partners (let's call it Action 17). Then, Gertrude and Fred switch partners (let's call it Action 18). Then, Fred and Gertrude switch partners. Then, Fred and Gertrude discuss something (let's call it Action 19). Then, Fred and Gertrude switch partners. Then, Alice and Bob switch partners (let's call it Action 20). Then, Bob and Alice switch partners (let's call it Action 21). Then, Action 18 repeats. Then, Gertrude and Fred discuss something (let's call it Action 22). Then, Gertrude and Fred switch partners. Then, nothing happens for a while. Then, Action 9 repeats. Then, Bob and Dave switch partners (let's call it Action 23). Then, Bob and Dave discuss something (let's call it Action 24). Then, Bob and Dave switch partners. Then, Dave and Claire switch partners (let's call it Action 25). Then, Dave and Claire discuss something (let's call it Action 26). Then, Dave and Claire switch partners. Then, Eve and Claire switch partners (let's call it Action 27). Then, Claire and Eve switch partners (let's call it Action 28). Then, nothing happens for a while. Then, Alice and Dave switch partners. Then, Alice and Dave discuss something (let's call it Action 29). Then, Alice and Dave switch partners. Then, Claire and Gertrude switch partners (let's call it Action 30). Then, Gertrude and Alice switch partners. Then, Alice and Gertrude switch partners. Then, Gertrude and Claire switch partners (let's call it Action 31). Then, Action 9 repeats. Then, Gertrude and Dave switch partners. Then, Action 5 repeats. Then, Dave and Gertrude switch partners. Then, Gertrude and Fred switch partners. Then, Action 17 repeats. Then, Action 8 repeats. Then, Gertrude and Eve switch partners (let's call it Action 32). Then, Eve and Fred switch partners. Then, Fred and Eve switch partners. Then, Eve and Gertrude switch partners (let's call it Action 33). Then, Fred and Bob switch partners (let's call it Action 34). Then, Fred and Bob discuss something (let's call it Action 35). Then, Action 34 repeats. Then, Action 25 repeats. Then, Dave and Claire discuss something. Then, Dave and Claire switch partners. Then, Gertrude and Claire switch partners. Then, Claire and Fred switch partners (let's call it Action 36). Then, Fred and Claire switch partners (let's call it Action 37). Then, Claire and Gertrude switch partners. Then, nothing happens for a while. Then, Action 9 repeats. Then, Alice and Dave switch partners. Then, Dave and Bob switch partners (let's call it Action 38). Then, Bob and Dave switch partners. Then, Dave and Alice switch partners. Then, Action 3 repeats. Then, Dave and Claire switch partners. Then, Claire and Dave switch partners (let's call it Action 39). Then, Dave and Alice switch partners. Then, Claire and Fred switch partners. Then, Fred and Claire switch partners. Then, Claire and Fred switch partners. Then, Claire and Fred discuss something (let's call it Action 40). Then, Claire and Fred switch partners. Then, Fred and Gertrude switch partners. Then, Gertrude and Alice switch partners. Then, Alice and Gertrude switch partners. Then, Gertrude and Fred switch partners. Then, nothing happens for a while. Then, Action 6 repeats. Then, Gertrude and Bob discuss something (let's call it Action 41). Then, Gertrude and Bob switch partners. Then, Eve and Alice switch partners (let's call it Action 42). Then, Alice and Eve switch partners (let's call it Action 43). Then, Fred and Dave switch partners (let's call it Action 44). Then, nothing happens for a while. Then, Alice and Eve switch partners. Then, Eve and Claire switch partners. Then, Claire and Eve switch partners. Then, Eve and Alice switch partners. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Action 43 repeats. Then, Eve and Alice switch partners. Then, Claire and Fred switch partners. Then, Action 11 repeats. Then, Eve and Fred switch partners. Then, Action 37 repeats. Then, nothing happens for a while. Then, Fred and Eve switch partners. Then, Eve and Bob switch partners (let's call it Action 45). Then, Bob and Eve switch partners (let's call it Action 46). Then, Action 1 repeats. Then, Fred and Gertrude switch partners. Then, Action 8 repeats. Then, Dave and Gertrude switch partners. Then, Gertrude and Fred switch partners. Then, Action 23 repeats. Then, Dave and Bob switch partners. Then, Action 9 repeats. Then, Eve and Alice switch partners. Then, Alice and Fred switch partners. Then, Fred and Alice switch partners. Then, Alice and Eve switch partners. Then, nothing happens for a while. Then, Dave and Eve switch partners. Then, Action 33 repeats. Then, Gertrude and Eve switch partners. Then, Action 12 repeats. Then, Action 43 repeats. Then, Action 42 repeats. Then, Gertrude and Alice switch partners. Then, Alice and Gertrude switch partners. Then, nothing happens for a while. Then, Action 9 repeats. Then, nothing happens for a while. Then, Gertrude and Dave switch partners. Then, Gertrude and Dave discuss something (let's call it Action 47). Then, Gertrude and Dave switch partners. Then, Action 9 repeats. Then, Fred and Claire switch partners. Then, Fred and Claire discuss something (let's call it Action 48). Then, Action 37 repeats. Then, Dave and Bob switch partners. Then, Bob and Dave switch partners. Then, Fred and Eve switch partners. Then, Action 1 repeats. Then, Alice and Dave switch partners. Then, Dave and Alice switch partners. Then, nothing happens for a while. Then, Bob and Eve switch partners. Then, Bob and Eve discuss something (let's call it Action 49). Then, Bob and Eve switch partners. Then, nothing happens for a while. Then, Fred and Alice switch partners. Then, Alice and Claire switch partners (let's call it Action 50). Then, Claire and Alice switch partners (let's call it Action 51). Then, Alice and Fred switch partners. Then, nothing happens for a while. Then, Fred and Eve switch partners. Then, Fred and Eve discuss something (let's call it Action 52). Then, Fred and Eve switch partners. Then, Bob and Gertrude switch partners. Then, Bob and Gertrude discuss something (let's call it Action 53). Then, Action 7 repeats. Then, Action 8 repeats. Then, Dave and Fred switch partners (let's call it Action 54). Then, Action 44 repeats. Then, Dave and Gertrude switch partners. Then, Gertrude and Dave switch partners. Then, Dave and Gertrude switch partners. Then, Fred and Claire switch partners. Then, Claire and Bob switch partners (let's call it Action 55). Then, Bob and Claire switch partners (let's call it Action 56). Then, Claire and Fred switch partners. Then, nothing happens for a while. Then, Action 3 repeats. Then, Dave and Alice switch partners. Then, Bob and Gertrude switch partners. Then, Gertrude and Bob switch partners. Then, Fred and Dave switch partners. Then, Fred and Dave discuss something (let's call it Action 57). Then, Fred and Dave switch partners. Then, Alice and Bob switch partners. Then, Alice and Bob discuss something (let's call it Action 58). Then, Alice and Bob switch partners. Then, Alice and Bob switch partners. Then, Dave and Claire switch partners. Then, Dave and Claire discuss something. Then, Dave and Claire switch partners. Then, Fred and Eve switch partners. Then, Eve and Fred switch partners. Then, Action 9 repeats. Then, Claire and Eve switch partners. Then, Claire and Eve discuss something (let's call it Action 59). Then, Claire and Eve switch partners. Then, nothing happens for a while. Then, nothing happens for a while. Then, Bob and Dave switch partners. Then, Dave and Bob switch partners. Then, Bob and Dave switch partners. Then, Dave and Gertrude switch partners. Then, Action 8 repeats. Then, Dave and Bob switch partners. Then, Action 37 repeats. Then, Fred and Claire discuss something. Then, Action 37 repeats. Then, nothing happens for a while. Then, Fred and Alice switch partners. Then, Alice and Fred switch partners. Then, nothing happens for a while. Then, Eve and Dave switch partners. Then, Action 10 repeats. Then, nothing happens for a while. Then, Gertrude and Claire switch partners. Then, Claire and Gertrude switch partners. Then, Claire and Fred switch partners. Then, Fred and Alice switch partners. Then, Action 16 repeats. Then, Fred and Claire switch partners. Then, Action 3 repeats. Then, Action 4 repeats. Then, Gertrude and Claire switch partners. Then, Action 30 repeats. Then, Bob and Fred switch partners (let's call it Action 60). Then, Fred and Bob switch partners. Then, Fred and Eve switch partners. Then, Eve and Alice switch partners. Then, Alice and Eve switch partners. Then, Eve and Fred switch partners. Then, Bob and Claire switch partners. Then, Claire and Bob switch partners. Then, Fred and Claire switch partners. Then, Claire and Fred switch partners. Then, nothing happens for a while. Then, Action 9 repeats. Then, Claire and Bob switch partners. Then, Bob and Claire switch partners. Then, Dave and Fred switch partners. Then, Dave and Fred discuss something (let's call it Action 61). Then, Dave and Fred switch partners. Then, Bob and Claire switch partners. Then, Bob and Claire discuss something (let's call it Action 62). Then, Bob and Claire switch partners. Then, Gertrude and Fred switch partners. Then, Action 22 repeats. Then, Gertrude and Fred switch partners. Then, nothing happens for a while. Then, Gertrude and Bob switch partners. Then, Gertrude and Alice switch partners. Then, Alice and Gertrude switch partners. Then, Eve and Bob switch partners. Then, Bob and Dave switch partners. Then, Action 38 repeats. Then, Bob and Eve switch partners. Then, Claire and Gertrude switch partners. Then, Gertrude and Claire switch partners. Then, Claire and Gertrude switch partners. Then, Claire and Gertrude discuss something (let's call it Action 63). Then, Claire and Gertrude switch partners. Then, Claire and Fred switch partners. Then, Claire and Fred discuss something. Then, Claire and Fred switch partners. Then, Dave and Gertrude switch partners. Then, Dave and Gertrude discuss something (let's call it Action 64). Then, Dave and Gertrude switch partners. Then, Gertrude and Alice switch partners. Then, Action 16 repeats. Then, Action 13 repeats. Then, Alice and Gertrude switch partners. Then, Claire and Dave switch partners. Then, Dave and Gertrude switch partners. Then, Gertrude and Dave switch partners. Then, Dave and Claire switch partners. Then, Action 14 repeats. Then, Gertrude and Fred switch partners. Then, Fred and Gertrude switch partners. Then, Gertrude and Alice switch partners. Then, Action 39 repeats. Then, Dave and Claire switch partners. Then, Claire and Eve switch partners. Then, Claire and Eve discuss something. Then, Action 28 repeats. Then, Fred and Alice switch partners. Then, Action 16 repeats. Then, Action 11 repeats. Then, Eve and Bob switch partners. Then, Bob and Eve switch partners. Then, Eve and Fred switch partners. Then, Claire and Eve switch partners. Then, Action 27 repeats. Then, Gertrude and Dave switch partners. Then, Dave and Bob switch partners. Then, Bob and Dave switch partners. Then, Dave and Gertrude switch partners. Then, nothing happens for a while. Then, nothing happens for a while. Then, Action 25 repeats. Then, Action 26 repeats. Then, Action 25 repeats. Then, Fred and Gertrude switch partners. Then, Gertrude and Fred switch partners. Then, Eve and Fred switch partners. Then, Action 11 repeats. Then, Alice and Eve switch partners. Then, Eve and Claire switch partners. Then, Claire and Eve switch partners. Then, Eve and Alice switch partners. Then, Action 13 repeats. Then, Fred and Alice discuss something (let's call it Action 65). Then, Fred and Alice switch partners. Then, Action 12 repeats. Then, Dave and Eve switch partners. Then, Action 37 repeats. Then, Claire and Fred switch partners. Then, Gertrude and Fred switch partners. Then, Fred and Gertrude switch partners. Then, Action 42 repeats. Then, Eve and Alice discuss something (let's call it Action 66). Then, Action 42 repeats. Then, Gertrude and Eve switch partners. Then, Gertrude and Eve discuss something (let's call it Action 67). Then, Gertrude and Eve switch partners. Then, Action 39 repeats. Then, Claire and Dave discuss something (let's call it Action 68). Then, Claire and Dave switch partners. Then, nothing happens for a while. Then, Action 5 repeats. Then, Dave and Gertrude discuss something. Then, Action 5 repeats. Then, Dave and Claire switch partners. Then, Claire and Dave switch partners. Then, nothing happens for a while. Then, Action 50 repeats. Then, Dave and Alice switch partners. Finally, Gertrude and Claire switch partners. At the end of the dance, Dave is dancing with\\nOptions:\\n(A) Izzi\\n(B) Sam\\n(C) Helga\\n(D) Jamie\\n(E) Rodrigo\\n(F) Ophelia\\n(G) Melissa\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}