{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cb8bf392-e941-54e6-9009-2b057382e471","task_key":"bbeh~5fshuffled~5fobjects--full--cb8bf392-e941-54e6-9009-2b057382e471","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are playing a game. At the…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are playing a game. At the start of the game, they are each holding a ball: Alice has a green ball, Bob has a brown ball, Claire has a purple ball, Dave has a pink ball, Eve has a white ball, Fred has a red ball, and Gertrude has a orange ball.\\nAs the game progresses, pairs of players trade balls. First, Eve and Dave swap balls. Then, Alice and Dave swap balls (let's call it Action 1). Then, Alice and Dave discuss something (let's call it Action 2). Then, Alice and Dave swap balls. Then, Action 1 repeats. Then, Alice and Dave discuss something. Then, Alice and Dave swap balls. Then, Gertrude and Fred swap balls (let's call it Action 3). Then, Fred and Gertrude swap balls (let's call it Action 4). Then, nothing happens for a while (let's call it Action 5). Then, Fred and Claire swap balls (let's call it Action 6). Then, Claire and Fred swap balls (let's call it Action 7). Then, Fred and Dave swap balls (let's call it Action 8). Then, Dave and Fred swap balls (let's call it Action 9). Then, Action 5 repeats. Then, Eve and Alice swap balls (let's call it Action 10). Then, Alice and Eve swap balls (let's call it Action 11). Then, Dave and Bob swap balls (let's call it Action 12). Then, Dave and Bob discuss something (let's call it Action 13). Then, Dave and Bob swap balls. Then, Action 5 repeats. Then, Gertrude and Bob swap balls (let's call it Action 14). Then, Bob and Eve swap balls (let's call it Action 15). Then, Eve and Bob swap balls (let's call it Action 16). Then, Bob and Gertrude swap balls (let's call it Action 17). Then, nothing happens for a while. Then, Bob and Alice swap balls (let's call it Action 18). Then, Bob and Alice discuss something (let's call it Action 19). Then, Action 18 repeats. Then, Action 16 repeats. Then, Action 15 repeats. Then, Action 5 repeats. Then, nothing happens for a while. Then, Dave and Gertrude swap balls (let's call it Action 20). Then, Action 14 repeats. Then, Bob and Gertrude swap balls. Then, Gertrude and Dave swap balls (let's call it Action 21). Then, nothing happens for a while. Then, Claire and Gertrude swap balls (let's call it Action 22). Then, Gertrude and Claire swap balls (let's call it Action 23). Then, Gertrude and Fred swap balls. Then, Action 8 repeats. Then, Action 9 repeats. Then, Fred and Gertrude swap balls. Then, Action 16 repeats. Then, Eve and Bob discuss something (let's call it Action 24). Then, Eve and Bob swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, Gertrude and Eve swap balls (let's call it Action 25). Then, Gertrude and Eve discuss something (let's call it Action 26). Then, Action 25 repeats. Then, Dave and Claire swap balls (let's call it Action 27). Then, Claire and Fred swap balls. Then, Action 6 repeats. Then, Claire and Dave swap balls (let's call it Action 28). Then, Gertrude and Eve swap balls. Then, Eve and Bob swap balls. Then, Action 15 repeats. Then, Eve and Gertrude swap balls (let's call it Action 29). Then, Claire and Fred swap balls. Then, Claire and Fred discuss something (let's call it Action 30). Then, Claire and Fred swap balls. Then, Eve and Claire swap balls (let's call it Action 31). Then, Claire and Eve swap balls (let's call it Action 32). Then, Action 5 repeats. Then, Claire and Eve swap balls. Then, Claire and Alice swap balls (let's call it Action 33). Then, Alice and Gertrude swap balls (let's call it Action 34). Then, Gertrude and Alice swap balls (let's call it Action 35). Then, Alice and Claire swap balls (let's call it Action 36). Then, Gertrude and Fred swap balls. Then, Fred and Gertrude swap balls. Then, Dave and Bob swap balls. Then, Dave and Bob discuss something. Then, Dave and Bob swap balls. Then, Action 35 repeats. Then, Gertrude and Alice discuss something (let's call it Action 37). Then, Action 35 repeats. Then, Alice and Eve swap balls. Then, Action 10 repeats. Then, nothing happens for a while. Then, Dave and Claire swap balls. Then, Claire and Bob swap balls (let's call it Action 38). Then, Bob and Claire swap balls (let's call it Action 39). Then, Claire and Dave swap balls. Then, Alice and Claire swap balls. Then, Claire and Alice swap balls. Then, Action 5 repeats. Then, Alice and Claire swap balls. Then, Claire and Bob swap balls. Then, Action 39 repeats. Then, Claire and Alice swap balls. Then, nothing happens for a while. Then, Alice and Bob swap balls (let's call it Action 40). Then, Bob and Alice swap balls. Then, nothing happens for a while. Then, Action 5 repeats. Then, Alice and Eve swap balls. Then, Eve and Bob swap balls. Then, Bob and Eve swap balls. Then, Action 10 repeats. Then, Dave and Alice swap balls (let's call it Action 41). Then, Alice and Dave swap balls. Then, Dave and Claire swap balls. Then, Claire and Dave swap balls. Then, Gertrude and Fred swap balls. Then, Fred and Alice swap balls (let's call it Action 42). Then, Alice and Fred swap balls (let's call it Action 43). Then, Action 4 repeats. Then, Bob and Eve swap balls. Then, Bob and Eve discuss something (let's call it Action 44). Then, Bob and Eve swap balls. Then, Gertrude and Fred swap balls. Then, Fred and Dave swap balls. Then, Action 9 repeats. Then, Action 4 repeats. Then, Fred and Claire swap balls. Then, Action 32 repeats. Then, Eve and Claire swap balls. Then, Action 7 repeats. Then, Action 28 repeats. Then, Dave and Fred swap balls. Then, Fred and Dave swap balls. Then, Dave and Claire swap balls. Then, Action 5 repeats. Then, Action 27 repeats. Then, Dave and Claire discuss something (let's call it Action 45). Then, Dave and Claire swap balls. Then, Fred and Eve swap balls (let's call it Action 46). Then, Eve and Alice swap balls. Then, Alice and Eve swap balls. Then, Eve and Fred swap balls (let's call it Action 47). Then, Dave and Bob swap balls. Then, Dave and Bob discuss something. Then, Dave and Bob swap balls. Then, Action 27 repeats. Then, nothing happens for a while. Then, Action 39 repeats. Then, Claire and Bob swap balls. Then, Action 5 repeats. Then, Fred and Eve swap balls. Then, Eve and Gertrude swap balls. Then, Gertrude and Eve swap balls. Then, Eve and Fred swap balls. Then, Bob and Dave swap balls (let's call it Action 48). Then, Dave and Alice swap balls. Then, Alice and Dave swap balls. Then, Dave and Bob swap balls. Then, Action 5 repeats. Then, Bob and Dave swap balls. Then, Dave and Bob swap balls. Then, Action 5 repeats. Then, Fred and Eve swap balls. Then, Fred and Eve discuss something (let's call it Action 49). Then, Action 46 repeats. Then, nothing happens for a while. Then, Fred and Eve swap balls. Then, Eve and Bob swap balls. Then, Bob and Eve swap balls. Then, Action 47 repeats. Then, Alice and Eve swap balls. Then, Eve and Alice swap balls. Then, Bob and Claire swap balls. Then, Claire and Alice swap balls. Then, Action 36 repeats. Then, Claire and Bob swap balls. Then, Action 41 repeats. Then, Dave and Alice discuss something (let's call it Action 50). Then, Dave and Alice swap balls. Then, Eve and Gertrude swap balls. Then, Eve and Gertrude discuss something (let's call it Action 51). Then, Eve and Gertrude swap balls. Then, nothing happens for a while. Then, Claire and Fred swap balls. Then, Action 6 repeats. Then, Gertrude and Alice swap balls. Then, Action 34 repeats. Then, Action 18 repeats. Then, Alice and Bob swap balls. Then, Eve and Bob swap balls. Then, Bob and Alice swap balls. Then, Alice and Bob swap balls. Then, Bob and Eve swap balls. Then, Bob and Fred swap balls (let's call it Action 52). Then, Action 18 repeats. Then, Alice and Bob swap balls. Then, Bob and Fred swap balls. Then, Bob and Fred discuss something (let's call it Action 53). Then, Bob and Fred swap balls. Then, Action 15 repeats. Then, Bob and Eve discuss something. Then, Bob and Eve swap balls. Then, nothing happens for a while. Then, Eve and Claire swap balls. Then, Claire and Dave swap balls. Then, Dave and Claire swap balls. Then, Action 32 repeats. Then, nothing happens for a while. Then, Alice and Dave swap balls. Then, Action 2 repeats. Then, Alice and Dave swap balls. Then, Bob and Alice swap balls. Then, Alice and Eve swap balls. Then, Eve and Alice swap balls. Then, Action 40 repeats. Then, Action 28 repeats. Then, Claire and Dave discuss something (let's call it Action 54). Then, Claire and Dave swap balls. Then, Gertrude and Eve swap balls. Then, Action 26 repeats. Then, Action 25 repeats. Then, Fred and Claire swap balls. Then, Action 7 repeats. Then, Action 14 repeats. Then, Action 17 repeats. Then, Dave and Bob swap balls. Then, Dave and Bob discuss something. Then, Dave and Bob swap balls. Then, Claire and Gertrude swap balls. Then, Claire and Gertrude discuss something (let's call it Action 55). Then, Claire and Gertrude swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, Gertrude and Fred swap balls. Then, Gertrude and Fred discuss something (let's call it Action 56). Then, Gertrude and Fred swap balls. Then, Bob and Gertrude swap balls. Then, Gertrude and Claire swap balls. Then, Claire and Gertrude swap balls. Then, Gertrude and Bob swap balls. Then, Action 16 repeats. Then, Action 24 repeats. Then, Eve and Bob swap balls. Then, Action 41 repeats. Then, Dave and Alice discuss something. Then, Action 41 repeats. Then, Fred and Claire swap balls. Then, Fred and Claire discuss something (let's call it Action 57). Then, Fred and Claire swap balls. Then, Claire and Alice swap balls. Then, Action 40 repeats. Then, Bob and Alice swap balls. Then, Alice and Claire swap balls. Then, Claire and Fred swap balls. Then, Action 30 repeats. Then, Claire and Fred swap balls. Then, nothing happens for a while. Then, Dave and Gertrude swap balls. Then, Action 21 repeats. Then, Action 14 repeats. Then, Bob and Gertrude swap balls. Then, Fred and Gertrude swap balls. Then, Gertrude and Fred swap balls. Then, Eve and Bob swap balls. Then, Bob and Eve swap balls. Then, Claire and Bob swap balls. Then, Claire and Bob discuss something (let's call it Action 58). Then, Claire and Bob swap balls. Then, Claire and Eve swap balls. Then, Eve and Claire swap balls. Then, Bob and Claire swap balls. Then, Bob and Claire discuss something (let's call it Action 59). Then, Action 39 repeats. Then, nothing happens for a while. Then, Claire and Dave swap balls. Then, Claire and Dave discuss something. Then, Claire and Dave swap balls. Then, nothing happens for a while. Then, Action 6 repeats. Then, Claire and Fred swap balls. Then, nothing happens for a while. Then, Alice and Bob swap balls. Then, Alice and Bob discuss something (let's call it Action 60). Then, Alice and Bob swap balls. Then, nothing happens for a while. Then, Action 7 repeats. Then, Fred and Alice swap balls. Then, Alice and Fred swap balls. Then, Fred and Claire swap balls. Then, Alice and Dave swap balls. Then, Alice and Dave discuss something. Then, Action 1 repeats. Then, Fred and Eve swap balls. Then, Eve and Fred swap balls. Then, Dave and Gertrude swap balls. Then, Gertrude and Fred swap balls. Then, Action 4 repeats. Then, Gertrude and Dave swap balls. Then, Action 8 repeats. Then, Eve and Alice swap balls. Finally, Dave and Gertrude swap balls. At the end of the game, Eve has the\\nOptions:\\n(A) green ball\\n(B) brown ball\\n(C) purple ball\\n(D) pink ball\\n(E) white ball\\n(F) red ball\\n(G) orange ball\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}