{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e0dc2a86-352c-5989-9df7-93fd4f1a4e26","task_key":"bbeh~5fshuffled~5fobjects--full--e0dc2a86-352c-5989-9df7-93fd4f1a4e26","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are playing a game. At the…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are playing a game. At the start of the game, they are each holding a ball: Alice has a purple ball, Bob has a black ball, Claire has a pink ball, Dave has a white ball, Eve has a green ball, Fred has a brown ball, and Gertrude has a red ball.\\nAs the game progresses, pairs of players trade balls. First, Eve and Claire swap balls. Then, nothing happens for a while (let's call it Action 1). Then, Fred and Alice swap balls (let's call it Action 2). Then, Alice and Fred swap balls (let's call it Action 3). Then, nothing happens for a while. Then, Bob and Fred swap balls (let's call it Action 4). Then, Fred and Alice swap balls. Then, Alice and Fred swap balls. Then, Fred and Bob swap balls (let's call it Action 5). Then, Alice and Eve swap balls (let's call it Action 6). Then, Eve and Alice swap balls (let's call it Action 7). Then, Dave and Fred swap balls (let's call it Action 8). Then, Fred and Dave swap balls (let's call it Action 9). Then, Alice and Eve swap balls. Then, Eve and Alice swap balls. Then, Bob and Alice swap balls (let's call it Action 10). Then, Bob and Alice discuss something (let's call it Action 11). Then, Action 10 repeats. Then, Gertrude and Fred swap balls (let's call it Action 12). Then, Fred and Claire swap balls (let's call it Action 13). Then, Claire and Fred swap balls (let's call it Action 14). Then, Fred and Gertrude swap balls (let's call it Action 15). Then, Dave and Eve swap balls (let's call it Action 16). Then, Eve and Dave swap balls (let's call it Action 17). Then, Action 16 repeats. Then, Eve and Bob swap balls (let's call it Action 18). Then, Bob and Eve swap balls (let's call it Action 19). Then, Action 17 repeats. Then, Eve and Fred swap balls (let's call it Action 20). Then, Eve and Fred discuss something (let's call it Action 21). Then, Action 20 repeats. Then, Bob and Claire swap balls (let's call it Action 22). Then, Claire and Fred swap balls. Then, Fred and Claire swap balls. Then, Claire and Bob swap balls (let's call it Action 23). Then, Claire and Alice swap balls (let's call it Action 24). Then, Claire and Alice discuss something (let's call it Action 25). Then, Claire and Alice swap balls. Then, nothing happens for a while. Then, Eve and Fred swap balls. Then, Eve and Fred discuss something. Then, Eve and Fred swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Claire swap balls (let's call it Action 26). Then, Claire and Dave swap balls (let's call it Action 27). Then, Fred and Gertrude swap balls. Then, Fred and Gertrude discuss something (let's call it Action 28). Then, Fred and Gertrude swap balls. Then, Claire and Fred swap balls. Then, Fred and Bob swap balls. Then, Bob and Fred swap balls. Then, Fred and Claire swap balls. Then, Alice and Eve swap balls. Then, Action 7 repeats. Then, Dave and Gertrude swap balls (let's call it Action 29). Then, Dave and Gertrude discuss something (let's call it Action 30). Then, Dave and Gertrude swap balls. Then, Action 10 repeats. Then, Bob and Alice discuss something. Then, Bob and Alice swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, Action 5 repeats. Then, Bob and Fred swap balls. Then, nothing happens for a while. Then, Claire and Gertrude swap balls (let's call it Action 31). Then, Gertrude and Claire swap balls (let's call it Action 32). Then, Bob and Fred swap balls. Then, Fred and Claire swap balls. Then, Claire and Fred swap balls. Then, Action 5 repeats. Then, Action 17 repeats. Then, Eve and Dave discuss something (let's call it Action 33). Then, Eve and Dave swap balls. Then, Fred and Gertrude swap balls. Then, Action 12 repeats. Then, Action 14 repeats. Then, Fred and Bob swap balls. Then, Bob and Fred swap balls. Then, Fred and Claire swap balls. Then, Action 1 repeats. Then, Eve and Fred swap balls. Then, Action 13 repeats. Then, Claire and Fred swap balls. Then, Fred and Eve swap balls (let's call it Action 34). Then, Alice and Dave swap balls (let's call it Action 35). Then, Dave and Alice swap balls (let's call it Action 36). Then, Bob and Dave swap balls (let's call it Action 37). Then, Bob and Dave discuss something (let's call it Action 38). Then, Bob and Dave swap balls. Then, Gertrude and Fred swap balls. Then, Action 13 repeats. Then, Action 14 repeats. Then, Fred and Gertrude swap balls. Then, Claire and Dave swap balls. Then, Dave and Claire swap balls. Then, nothing happens for a while. Then, Action 1 repeats. Then, Dave and Bob swap balls (let's call it Action 39). Then, Dave and Bob discuss something (let's call it Action 40). Then, Dave and Bob swap balls. Then, Fred and Eve swap balls. Then, Fred and Eve discuss something (let's call it Action 41). Then, Fred and Eve swap balls. Then, Action 1 repeats. Then, Bob and Claire swap balls. Then, Claire and Bob swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, Gertrude and Eve swap balls (let's call it Action 42). Then, Gertrude and Eve discuss something (let's call it Action 43). Then, Gertrude and Eve swap balls. Then, Action 7 repeats. Then, Alice and Eve swap balls. Then, Alice and Gertrude swap balls (let's call it Action 44). Then, Gertrude and Alice swap balls (let's call it Action 45). Then, Bob and Gertrude swap balls (let's call it Action 46). Then, nothing happens for a while. Then, Action 26 repeats. Then, Dave and Claire discuss something (let's call it Action 47). Then, Dave and Claire swap balls. Then, Eve and Bob swap balls. Then, Eve and Bob discuss something (let's call it Action 48). Then, Eve and Bob swap balls. Then, Alice and Dave swap balls. Then, Dave and Eve swap balls. Then, Eve and Dave swap balls. Then, Dave and Alice swap balls. Then, Action 6 repeats. Then, Eve and Fred swap balls. Then, Fred and Eve swap balls. Then, Action 7 repeats. Then, Dave and Claire swap balls. Then, Action 27 repeats. Then, Eve and Gertrude swap balls (let's call it Action 49). Then, Gertrude and Fred swap balls. Then, Action 15 repeats. Then, Action 42 repeats. Then, nothing happens for a while. Then, Bob and Claire swap balls. Then, Claire and Eve swap balls (let's call it Action 50). Then, Eve and Claire swap balls (let's call it Action 51). Then, Claire and Bob swap balls. Then, nothing happens for a while. Then, Action 51 repeats. Then, Eve and Claire discuss something (let's call it Action 52). Then, Eve and Claire swap balls. Then, Fred and Gertrude swap balls. Then, Fred and Gertrude discuss something. Then, Fred and Gertrude swap balls. Then, Eve and Dave swap balls. Then, Action 26 repeats. Then, Action 27 repeats. Then, Dave and Eve swap balls. Then, Action 44 repeats. Then, Gertrude and Dave swap balls (let's call it Action 53). Then, Dave and Gertrude swap balls. Then, Gertrude and Alice swap balls. Then, Dave and Bob swap balls. Then, Action 37 repeats. Then, Gertrude and Alice swap balls. Then, Gertrude and Alice discuss something (let's call it Action 54). Then, Action 45 repeats. Then, Claire and Dave swap balls. Then, Claire and Dave discuss something (let's call it Action 55). Then, Claire and Dave swap balls. Then, Action 36 repeats. Then, Dave and Alice discuss something (let's call it Action 56). Then, Dave and Alice swap balls. Then, Action 51 repeats. Then, Claire and Eve swap balls. Then, Bob and Fred swap balls. Then, Fred and Dave swap balls. Then, Dave and Fred swap balls. Then, Fred and Bob swap balls. Then, Action 1 repeats. Then, nothing happens for a while. Then, Bob and Fred swap balls. Then, Bob and Fred discuss something (let's call it Action 57). Then, Action 4 repeats. Then, Claire and Dave swap balls. Then, Dave and Alice swap balls. Then, Alice and Dave swap balls. Then, Dave and Claire swap balls. Then, Fred and Bob swap balls. Then, Action 10 repeats. Then, Alice and Bob swap balls (let's call it Action 58). Then, Action 4 repeats. Then, Alice and Eve swap balls. Then, Eve and Claire swap balls. Then, Action 50 repeats. Then, Eve and Alice swap balls. Then, Action 36 repeats. Then, Action 35 repeats. Then, Gertrude and Bob swap balls (let's call it Action 59). Then, Bob and Gertrude swap balls. Then, nothing happens for a while. Then, Alice and Dave swap balls. Then, Claire and Dave swap balls. Then, Action 16 repeats. Then, Action 17 repeats. Then, Dave and Claire swap balls. Then, Eve and Alice swap balls. Then, Alice and Dave swap balls. Then, Dave and Alice swap balls. Then, Action 6 repeats. Then, Dave and Gertrude swap balls. Then, Action 59 repeats. Then, Action 46 repeats. Then, Gertrude and Dave swap balls. Then, Alice and Fred swap balls. Then, Fred and Dave swap balls. Then, Dave and Fred swap balls. Then, Fred and Alice swap balls. Then, Action 1 repeats. Then, Eve and Alice swap balls. Then, Eve and Alice discuss something (let's call it Action 60). Then, Eve and Alice swap balls. Then, nothing happens for a while. Then, Claire and Dave swap balls. Then, Dave and Eve swap balls. Then, Eve and Dave swap balls. Then, Dave and Claire swap balls. Then, nothing happens for a while. Then, Action 26 repeats. Then, Dave and Claire discuss something. Then, Action 26 repeats. Then, nothing happens for a while. Then, Gertrude and Bob swap balls. Then, Action 46 repeats. Then, Dave and Bob swap balls. Then, Dave and Bob discuss something. Then, Dave and Bob swap balls. Then, Action 23 repeats. Then, Bob and Fred swap balls. Then, Action 5 repeats. Then, Bob and Claire swap balls. Then, Eve and Alice swap balls. Then, Alice and Claire swap balls (let's call it Action 61). Then, Claire and Alice swap balls. Then, Action 6 repeats. Then, Eve and Alice swap balls. Then, Alice and Eve swap balls. Then, Bob and Alice swap balls. Then, Action 3 repeats. Then, Action 2 repeats. Then, Alice and Bob swap balls. Then, Claire and Bob swap balls. Then, Action 4 repeats. Then, Fred and Bob swap balls. Then, Bob and Claire swap balls. Then, Action 5 repeats. Then, Bob and Fred swap balls. Then, nothing happens for a while. Then, Action 51 repeats. Then, Claire and Eve swap balls. Then, Action 51 repeats. Then, Action 23 repeats. Then, Bob and Claire swap balls. Then, Claire and Eve swap balls. Then, Fred and Gertrude swap balls. Then, Fred and Gertrude discuss something. Then, Fred and Gertrude swap balls. Then, Action 15 repeats. Then, Gertrude and Fred swap balls. Then, Action 19 repeats. Then, Action 49 repeats. Then, Action 42 repeats. Then, Eve and Bob swap balls. Then, Fred and Claire swap balls. Then, Alice and Dave swap balls. Then, Alice and Dave discuss something (let's call it Action 62). Then, Alice and Dave swap balls. Then, nothing happens for a while. Then, Fred and Alice swap balls. Then, Alice and Fred swap balls. Then, Action 7 repeats. Then, Alice and Eve swap balls. Then, Bob and Alice swap balls. Then, Alice and Bob swap balls. Then, Bob and Eve swap balls. Then, Bob and Eve discuss something (let's call it Action 63). Then, Bob and Eve swap balls. Then, Dave and Fred swap balls. Then, Dave and Fred discuss something (let's call it Action 64). Then, Dave and Fred swap balls. Then, Action 27 repeats. Then, Dave and Alice swap balls. Then, Alice and Dave swap balls. Then, Dave and Claire swap balls. Then, Alice and Bob swap balls. Then, Bob and Gertrude swap balls. Then, Gertrude and Bob swap balls. Then, Bob and Alice swap balls. Then, Action 1 repeats. Then, nothing happens for a while. Then, Dave and Eve swap balls. Then, Dave and Eve discuss something (let's call it Action 65). Then, Dave and Eve swap balls. Then, Action 50 repeats. Then, Claire and Eve discuss something (let's call it Action 66). Then, Claire and Eve swap balls. Then, Claire and Dave swap balls. Then, Dave and Claire swap balls. Then, Gertrude and Dave swap balls. Then, Action 29 repeats. Then, Gertrude and Dave swap balls. Then, Action 36 repeats. Then, Alice and Dave swap balls. Then, Dave and Gertrude swap balls. Then, Claire and Bob swap balls. Then, Claire and Bob discuss something (let's call it Action 67). Then, Claire and Bob swap balls. Then, Eve and Bob swap balls. Then, Bob and Alice swap balls. Then, Action 58 repeats. Then, Bob and Eve swap balls. Then, Action 1 repeats. Then, Claire and Alice swap balls. Then, Action 25 repeats. Then, Claire and Alice swap balls. Then, nothing happens for a while. Then, Dave and Gertrude swap balls. Then, Gertrude and Dave swap balls. Then, Gertrude and Claire swap balls. Then, Dave and Alice swap balls. Finally, Eve and Alice swap balls. At the end of the game, Dave has the\\nOptions:\\n(A) purple ball\\n(B) black ball\\n(C) pink ball\\n(D) white ball\\n(E) green ball\\n(F) brown ball\\n(G) red ball\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}