{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"515c737d-1fee-5c6e-b781-deabc5723420","task_key":"bbeh~5fshuffled~5fobjects--full--515c737d-1fee-5c6e-b781-deabc5723420","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are dancers at a square dance.…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are dancers at a square dance. At the start of a song, they each have a partner: Alice is dancing with Helga, Bob is dancing with Patrick, Claire is dancing with Jamie, Dave is dancing with Karl, Eve is dancing with Ophelia, Fred is dancing with Izzi, and Gertrude is dancing with Melissa.\\nThroughout the song, the dancers often trade partners. First, Gertrude and Claire switch partners. Then, Gertrude and Dave switch partners (let's call it Action 1). Then, Dave and Gertrude switch partners (let's call it Action 2). Then, Eve and Alice switch partners (let's call it Action 3). Then, Alice and Fred switch partners (let's call it Action 4). Then, Fred and Alice switch partners (let's call it Action 5). Then, Alice and Eve switch partners (let's call it Action 6). Then, Gertrude and Eve switch partners (let's call it Action 7). Then, Gertrude and Eve discuss something (let's call it Action 8). Then, Action 7 repeats. Then, Bob and Claire switch partners (let's call it Action 9). Then, Claire and Bob switch partners (let's call it Action 10). Then, Dave and Claire switch partners (let's call it Action 11). Then, Claire and Fred switch partners (let's call it Action 12). Then, Fred and Claire switch partners (let's call it Action 13). Then, Claire and Dave switch partners (let's call it Action 14). Then, Fred and Claire switch partners. Then, Claire and Fred switch partners. Then, Eve and Fred switch partners (let's call it Action 15). Then, Eve and Fred discuss something (let's call it Action 16). Then, Action 15 repeats. Then, Bob and Claire switch partners. Then, Claire and Bob switch partners. Then, Alice and Eve switch partners. Then, Alice and Eve discuss something (let's call it Action 17). Then, Alice and Eve switch partners. Then, Bob and Eve switch partners (let's call it Action 18). Then, Bob and Eve discuss something (let's call it Action 19). Then, Action 18 repeats. Then, Bob and Claire switch partners. Then, Claire and Alice switch partners (let's call it Action 20). Then, Alice and Claire switch partners (let's call it Action 21). Then, Action 10 repeats. Then, Eve and Dave switch partners (let's call it Action 22). Then, Eve and Dave discuss something (let's call it Action 23). Then, Eve and Dave switch partners. Then, Eve and Claire switch partners (let's call it Action 24). Then, Eve and Claire discuss something (let's call it Action 25). Then, Action 24 repeats. Then, Eve and Bob switch partners (let's call it Action 26). Then, Eve and Bob discuss something (let's call it Action 27). Then, Action 26 repeats. Then, Action 2 repeats. Then, Dave and Gertrude discuss something (let's call it Action 28). Then, Dave and Gertrude switch partners. Then, Gertrude and Eve switch partners. Then, Eve and Gertrude switch partners (let's call it Action 29). Then, Alice and Bob switch partners (let's call it Action 30). Then, Alice and Bob discuss something (let's call it Action 31). Then, Alice and Bob switch partners. Then, Dave and Bob switch partners (let's call it Action 32). Then, Dave and Bob discuss something (let's call it Action 33). Then, Action 32 repeats. Then, nothing happens for a while (let's call it Action 34). Then, Eve and Bob switch partners. Then, Action 18 repeats. Then, Action 5 repeats. Then, Fred and Alice discuss something (let's call it Action 35). Then, Action 5 repeats. Then, Action 34 repeats. Then, Fred and Alice switch partners. Then, Action 4 repeats. Then, Action 26 repeats. Then, Bob and Dave switch partners (let's call it Action 36). Then, Dave and Bob switch partners. Then, Action 18 repeats. Then, Eve and Gertrude switch partners. Then, Gertrude and Eve switch partners. Then, Gertrude and Eve switch partners. Then, Gertrude and Eve discuss something. Then, Gertrude and Eve switch partners. Then, Gertrude and Fred switch partners (let's call it Action 37). Then, Fred and Claire switch partners. Then, Claire and Fred switch partners. Then, Fred and Gertrude switch partners (let's call it Action 38). Then, Alice and Dave switch partners (let's call it Action 39). Then, Dave and Alice switch partners (let's call it Action 40). Then, Gertrude and Fred switch partners. Then, Gertrude and Fred discuss something (let's call it Action 41). Then, Gertrude and Fred switch partners. Then, nothing happens for a while. Then, Alice and Claire switch partners. Then, Alice and Claire discuss something (let's call it Action 42). Then, Alice and Claire switch partners. Then, Eve and Alice switch partners. Then, Action 12 repeats. Then, Fred and Eve switch partners (let's call it Action 43). Then, Eve and Fred switch partners. Then, Fred and Claire switch partners. Then, Alice and Claire switch partners. Then, Claire and Alice switch partners. Then, Gertrude and Fred switch partners. Then, Fred and Gertrude switch partners. Then, Claire and Gertrude switch partners (let's call it Action 44). Then, Claire and Gertrude discuss something (let's call it Action 45). Then, Claire and Gertrude switch partners. Then, Fred and Bob switch partners (let's call it Action 46). Then, Fred and Bob discuss something (let's call it Action 47). Then, Action 46 repeats. Then, Bob and Gertrude switch partners (let's call it Action 48). Then, Gertrude and Alice switch partners (let's call it Action 49). Then, Alice and Gertrude switch partners (let's call it Action 50). Then, Gertrude and Bob switch partners (let's call it Action 51). Then, Gertrude and Alice switch partners. Then, Gertrude and Alice discuss something (let's call it Action 52). Then, Action 49 repeats. Then, nothing happens for a while. Then, Dave and Eve switch partners (let's call it Action 53). Then, Action 22 repeats. Then, Claire and Bob switch partners. Then, Bob and Dave switch partners. Then, Action 32 repeats. Then, Action 9 repeats. Then, nothing happens for a while. Then, Action 15 repeats. Then, Fred and Dave switch partners (let's call it Action 54). Then, Dave and Fred switch partners (let's call it Action 55). Then, Fred and Eve switch partners. Then, Fred and Gertrude switch partners. Then, Gertrude and Fred switch partners. Then, nothing happens for a while. Then, Bob and Alice switch partners (let's call it Action 56). Then, Alice and Fred switch partners. Then, Fred and Alice switch partners. Then, Alice and Bob switch partners. Then, Eve and Alice switch partners. Then, Eve and Alice discuss something (let's call it Action 57). Then, Eve and Alice switch partners. Then, Eve and Claire switch partners. Then, Eve and Claire discuss something. Then, Eve and Claire switch partners. Then, Action 26 repeats. Then, Bob and Dave switch partners. Then, Action 32 repeats. Then, Bob and Eve switch partners. Then, nothing happens for a while. Then, nothing happens for a while. Then, Bob and Claire switch partners. Then, Action 44 repeats. Then, Gertrude and Claire switch partners (let's call it Action 58). Then, Claire and Bob switch partners. Then, Action 38 repeats. Then, Fred and Gertrude discuss something (let's call it Action 59). Then, Fred and Gertrude switch partners. Then, Alice and Bob switch partners. Then, Bob and Alice switch partners. Then, Action 34 repeats. Then, nothing happens for a while. Then, Bob and Alice switch partners. Then, Bob and Alice discuss something (let's call it Action 60). Then, Bob and Alice switch partners. Then, Action 1 repeats. Then, Dave and Eve switch partners. Then, Eve and Dave switch partners. Then, Dave and Gertrude switch partners. Then, Gertrude and Eve switch partners. Then, Gertrude and Eve discuss something. Then, Gertrude and Eve switch partners. Then, Action 3 repeats. Then, Eve and Alice discuss something. Then, Action 3 repeats. Then, Action 34 repeats. Then, Eve and Gertrude switch partners. Then, Gertrude and Eve switch partners. Then, Gertrude and Fred switch partners. Then, Fred and Gertrude switch partners. Then, Claire and Dave switch partners. Then, Dave and Gertrude switch partners. Then, Action 1 repeats. Then, Action 11 repeats. Then, Fred and Claire switch partners. Then, Fred and Claire discuss something (let's call it Action 61). Then, Fred and Claire switch partners. Then, Dave and Fred switch partners. Then, Dave and Fred discuss something (let's call it Action 62). Then, Dave and Fred switch partners. Then, Claire and Bob switch partners. Then, Bob and Alice switch partners. Then, Alice and Bob switch partners. Then, Bob and Claire switch partners. Then, Action 34 repeats. Then, Dave and Eve switch partners. Then, Action 3 repeats. Then, Alice and Eve switch partners. Then, Eve and Dave switch partners. Then, nothing happens for a while. Then, Alice and Claire switch partners. Then, Alice and Claire discuss something. Then, Action 21 repeats. Then, Alice and Dave switch partners. Then, Dave and Alice switch partners. Then, Fred and Claire switch partners. Then, Action 12 repeats. Then, Bob and Fred switch partners (let's call it Action 63). Then, Bob and Fred discuss something (let's call it Action 64). Then, Bob and Fred switch partners. Then, Action 49 repeats. Then, Alice and Eve switch partners. Then, Eve and Alice switch partners. Then, Alice and Gertrude switch partners. Then, Gertrude and Bob switch partners. Then, Bob and Gertrude switch partners. Then, Claire and Eve switch partners (let's call it Action 65). Then, Eve and Gertrude switch partners. Then, Gertrude and Eve switch partners. Then, Eve and Claire switch partners. Then, Action 46 repeats. Then, Action 14 repeats. Then, Claire and Dave discuss something (let's call it Action 66). Then, Claire and Dave switch partners. Then, Action 6 repeats. Then, Eve and Alice switch partners. Then, Action 51 repeats. Then, Bob and Alice switch partners. Then, Alice and Bob switch partners. Then, Bob and Gertrude switch partners. Then, nothing happens for a while. Then, Dave and Alice switch partners. Then, Alice and Dave switch partners. Then, Bob and Gertrude switch partners. Then, Gertrude and Eve switch partners. Then, Eve and Gertrude switch partners. Then, Action 51 repeats. Then, Action 13 repeats. Then, Claire and Gertrude switch partners. Then, Action 58 repeats. Then, Claire and Fred switch partners. Then, Bob and Fred switch partners. Then, Bob and Fred discuss something. Then, Action 63 repeats. Then, Gertrude and Dave switch partners. Then, Dave and Alice switch partners. Then, Alice and Dave switch partners. Then, Dave and Gertrude switch partners. Then, nothing happens for a while. Then, Action 34 repeats. Then, nothing happens for a while. Then, Action 39 repeats. Then, Dave and Alice switch partners. Then, Alice and Fred switch partners. Then, Alice and Fred discuss something (let's call it Action 67). Then, Action 4 repeats. Then, nothing happens for a while. Then, Claire and Bob switch partners. Then, Claire and Bob discuss something (let's call it Action 68). Then, Claire and Bob switch partners. Then, Action 18 repeats. Then, Eve and Bob switch partners. Then, Action 53 repeats. Then, Eve and Bob switch partners. Then, Bob and Eve switch partners. Then, Action 22 repeats. Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Claire switch partners. Then, Action 14 repeats. Then, Bob and Claire switch partners. Then, Action 10 repeats. Then, Action 18 repeats. Then, Eve and Bob switch partners. Then, Dave and Alice switch partners. Then, Gertrude and Dave switch partners. Then, Action 2 repeats. Then, Claire and Alice switch partners. Then, Claire and Alice discuss something (let's call it Action 69). Then, Action 20 repeats. Then, Action 54 repeats. Then, Dave and Alice switch partners. Then, Alice and Dave switch partners. Then, Dave and Fred switch partners. Then, Gertrude and Dave switch partners. Then, Dave and Gertrude switch partners. Then, Action 55 repeats. Then, Fred and Gertrude switch partners. Then, Gertrude and Fred switch partners. Then, Fred and Dave switch partners. Then, Action 20 repeats. Then, Action 50 repeats. Then, Gertrude and Alice switch partners. Then, Alice and Claire switch partners. Then, nothing happens for a while. Then, Alice and Dave switch partners. Then, Dave and Claire switch partners. Then, Claire and Dave switch partners. Then, Action 40 repeats. Then, Alice and Fred switch partners. Then, Fred and Bob switch partners. Then, Bob and Fred switch partners. Then, Fred and Alice switch partners. Then, Action 34 repeats. Then, Gertrude and Bob switch partners. Then, Bob and Alice switch partners. Then, Alice and Bob switch partners. Then, Bob and Gertrude switch partners. Then, nothing happens for a while. Then, nothing happens for a while. Then, Action 34 repeats. Then, Action 34 repeats. Then, Claire and Fred switch partners. Then, Fred and Claire switch partners. Then, Action 54 repeats. Then, Dave and Claire switch partners. Then, Claire and Dave switch partners. Then, Dave and Fred switch partners. Then, Bob and Gertrude switch partners. Then, Gertrude and Bob switch partners. Then, Action 34 repeats. Then, Action 34 repeats. Then, Action 13 repeats. Then, Claire and Fred switch partners. Then, Alice and Claire switch partners. Then, Dave and Bob switch partners. Finally, Gertrude and Fred switch partners. At the end of the dance, Dave is dancing with\\nOptions:\\n(A) Helga\\n(B) Patrick\\n(C) Jamie\\n(D) Karl\\n(E) Ophelia\\n(F) Izzi\\n(G) Melissa\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}