{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"70dc6b97-4249-5090-b41a-47ba86ec45c1","task_key":"bbeh~5fshuffled~5fobjects--full--70dc6b97-4249-5090-b41a-47ba86ec45c1","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are playing a game. At the…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are playing a game. At the start of the game, they are each holding a ball: Alice has a green ball, Bob has a brown ball, Claire has a orange ball, Dave has a red ball, Eve has a blue ball, Fred has a black ball, and Gertrude has a yellow ball.\\nAs the game progresses, pairs of players trade balls. First, Eve and Dave swap balls. Then, nothing happens for a while (let's call it Action 1). Then, Alice and Dave swap balls (let's call it Action 2). Then, Alice and Dave discuss something (let's call it Action 3). Then, Alice and Dave swap balls. Then, Claire and Gertrude swap balls (let's call it Action 4). Then, Gertrude and Claire swap balls (let's call it Action 5). Then, Action 1 repeats. Then, Eve and Fred swap balls (let's call it Action 6). Then, Fred and Alice swap balls (let's call it Action 7). Then, Alice and Fred swap balls (let's call it Action 8). Then, Fred and Eve swap balls (let's call it Action 9). Then, Gertrude and Claire swap balls. Then, Claire and Eve swap balls (let's call it Action 10). Then, Eve and Claire swap balls (let's call it Action 11). Then, Claire and Gertrude swap balls. Then, Alice and Dave swap balls. Then, Dave and Alice swap balls (let's call it Action 12). Then, Claire and Fred swap balls (let's call it Action 13). Then, Fred and Claire swap balls (let's call it Action 14). Then, Gertrude and Bob swap balls (let's call it Action 15). Then, Bob and Alice swap balls (let's call it Action 16). Then, Alice and Bob swap balls (let's call it Action 17). Then, Bob and Gertrude swap balls (let's call it Action 18). Then, Eve and Bob swap balls (let's call it Action 19). Then, Eve and Bob discuss something (let's call it Action 20). Then, Action 19 repeats. Then, Alice and Eve swap balls (let's call it Action 21). Then, Alice and Eve discuss something (let's call it Action 22). Then, Alice and Eve swap balls. Then, Claire and Bob swap balls (let's call it Action 23). Then, Claire and Bob discuss something (let's call it Action 24). Then, Claire and Bob swap balls. Then, Claire and Eve swap balls. Then, Claire and Eve discuss something (let's call it Action 25). Then, Claire and Eve swap balls. Then, Dave and Claire swap balls (let's call it Action 26). Then, Dave and Claire discuss something (let's call it Action 27). Then, Dave and Claire swap balls. Then, Fred and Gertrude swap balls (let's call it Action 28). Then, Fred and Gertrude discuss something (let's call it Action 29). Then, Fred and Gertrude swap balls. Then, Alice and Gertrude swap balls (let's call it Action 30). Then, Alice and Gertrude discuss something (let's call it Action 31). Then, Alice and Gertrude swap balls. Then, Action 19 repeats. Then, Bob and Eve swap balls (let's call it Action 32). Then, Action 1 repeats. Then, Alice and Fred swap balls. Then, Action 7 repeats. Then, Action 30 repeats. Then, Alice and Gertrude discuss something. Then, Alice and Gertrude swap balls. Then, nothing happens for a while. Then, Gertrude and Eve swap balls (let's call it Action 33). Then, nothing happens for a while. Then, Claire and Fred swap balls. Then, Claire and Fred discuss something (let's call it Action 34). Then, Claire and Fred swap balls. Then, Action 1 repeats. Then, nothing happens for a while. Then, Alice and Claire swap balls (let's call it Action 35). Then, Alice and Claire discuss something (let's call it Action 36). Then, Action 35 repeats. Then, Eve and Dave swap balls (let's call it Action 37). Then, Eve and Dave discuss something (let's call it Action 38). Then, Eve and Dave swap balls. Then, Dave and Claire swap balls. Then, Dave and Claire discuss something. Then, Dave and Claire swap balls. Then, Claire and Gertrude swap balls. Then, Claire and Gertrude discuss something (let's call it Action 39). Then, Claire and Gertrude swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, Eve and Fred swap balls. Then, Fred and Eve swap balls. Then, Alice and Dave swap balls. Then, Dave and Alice swap balls. Then, nothing happens for a while. Then, Bob and Dave swap balls (let's call it Action 40). Then, Dave and Alice swap balls. Then, Alice and Dave swap balls. Then, Dave and Bob swap balls (let's call it Action 41). Then, Action 1 repeats. Then, Action 1 repeats. Then, nothing happens for a while. Then, Dave and Alice swap balls. Then, Alice and Dave swap balls. Then, Alice and Fred swap balls. Then, Alice and Fred discuss something (let's call it Action 42). Then, Alice and Fred swap balls. Then, Alice and Bob swap balls. Then, Alice and Bob discuss something (let's call it Action 43). Then, Alice and Bob swap balls. Then, Fred and Bob swap balls (let's call it Action 44). Then, Bob and Fred swap balls (let's call it Action 45). Then, Dave and Fred swap balls (let's call it Action 46). Then, Dave and Fred discuss something (let's call it Action 47). Then, Dave and Fred swap balls. Then, Alice and Bob swap balls. Then, Bob and Claire swap balls (let's call it Action 48). Then, Claire and Bob swap balls. Then, Action 16 repeats. Then, Bob and Fred swap balls. Then, Claire and Gertrude swap balls. Then, Gertrude and Fred swap balls (let's call it Action 49). Then, Action 28 repeats. Then, Action 5 repeats. Then, nothing happens for a while. Then, Dave and Fred swap balls. Then, Fred and Gertrude swap balls. Then, Gertrude and Fred swap balls. Then, Fred and Dave swap balls (let's call it Action 50). Then, Gertrude and Claire swap balls. Then, Claire and Gertrude swap balls. Then, Eve and Fred swap balls. Then, Fred and Eve swap balls. Then, nothing happens for a while. Then, Fred and Dave swap balls. Then, Fred and Dave discuss something (let's call it Action 51). Then, Fred and Dave swap balls. Then, Alice and Claire swap balls. Then, Action 36 repeats. Then, Alice and Claire swap balls. Then, Action 1 repeats. Then, Dave and Claire swap balls. Then, Claire and Eve swap balls. Then, Eve and Claire swap balls. Then, Claire and Dave swap balls (let's call it Action 52). Then, Gertrude and Eve swap balls. Then, Eve and Claire swap balls. Then, Claire and Eve swap balls. Then, Eve and Gertrude swap balls (let's call it Action 53). Then, nothing happens for a while. Then, Claire and Eve swap balls. Then, Eve and Fred swap balls. Then, Fred and Eve swap balls. Then, Eve and Claire swap balls. Then, Eve and Bob swap balls. Then, Action 45 repeats. Then, Fred and Bob swap balls. Then, Bob and Eve swap balls. Then, Action 19 repeats. Then, Eve and Bob discuss something. Then, Eve and Bob swap balls. Then, Gertrude and Alice swap balls (let's call it Action 54). Then, Alice and Gertrude swap balls. Then, Fred and Claire swap balls. Then, Claire and Fred swap balls. Then, Eve and Fred swap balls. Then, Fred and Eve swap balls. Then, Alice and Bob swap balls. Then, Bob and Alice swap balls. Then, Action 32 repeats. Then, Eve and Fred swap balls. Then, Fred and Eve swap balls. Then, Action 19 repeats. Then, Bob and Fred swap balls. Then, Fred and Bob swap balls. Then, Fred and Eve swap balls. Then, Eve and Fred swap balls. Then, Bob and Eve swap balls. Then, Eve and Dave swap balls. Then, Dave and Eve swap balls (let's call it Action 55). Then, Eve and Bob swap balls. Then, Eve and Claire swap balls. Then, Claire and Eve swap balls. Then, Dave and Claire swap balls. Then, Claire and Dave swap balls. Then, Action 21 repeats. Then, Alice and Eve discuss something. Then, Alice and Eve swap balls. Then, Action 1 repeats. Then, Gertrude and Fred swap balls. Then, Gertrude and Fred discuss something (let's call it Action 56). Then, Gertrude and Fred swap balls. Then, Action 13 repeats. Then, Claire and Fred discuss something. Then, Claire and Fred swap balls. Then, Claire and Bob swap balls. Then, Bob and Eve swap balls. Then, Eve and Bob swap balls. Then, Action 48 repeats. Then, Eve and Alice swap balls (let's call it Action 57). Then, Alice and Eve swap balls. Then, Fred and Dave swap balls. Then, Dave and Fred swap balls. Then, Action 23 repeats. Then, Bob and Claire swap balls. Then, nothing happens for a while. Then, Fred and Eve swap balls. Then, Fred and Eve discuss something (let's call it Action 58). Then, Fred and Eve swap balls. Then, Fred and Claire swap balls. Then, Claire and Alice swap balls (let's call it Action 59). Then, Alice and Claire swap balls. Then, Claire and Fred swap balls. Then, Eve and Dave swap balls. Then, Action 38 repeats. Then, Eve and Dave swap balls. Then, nothing happens for a while. Then, Action 26 repeats. Then, Claire and Dave swap balls. Then, Eve and Fred swap balls. Then, Action 9 repeats. Then, Eve and Gertrude swap balls. Then, Action 15 repeats. Then, Bob and Gertrude swap balls. Then, Gertrude and Eve swap balls. Then, Action 1 repeats. Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Gertrude swap balls (let's call it Action 60). Then, Gertrude and Dave swap balls (let's call it Action 61). Then, Claire and Fred swap balls. Then, Dave and Bob swap balls. Then, Bob and Gertrude swap balls. Then, Gertrude and Bob swap balls. Then, Bob and Dave swap balls. Then, nothing happens for a while. Then, Action 46 repeats. Then, Fred and Dave swap balls. Then, Action 30 repeats. Then, Alice and Gertrude discuss something. Then, Alice and Gertrude swap balls. Then, Action 8 repeats. Then, Fred and Alice swap balls. Then, Action 1 repeats. Then, nothing happens for a while. Then, Eve and Bob swap balls. Then, Eve and Bob discuss something. Then, Eve and Bob swap balls. Then, Eve and Fred swap balls. Then, Eve and Fred discuss something (let's call it Action 62). Then, Eve and Fred swap balls. Then, Action 1 repeats. Then, Gertrude and Alice swap balls. Then, Alice and Gertrude swap balls. Then, Gertrude and Dave swap balls. Then, Dave and Alice swap balls. Then, Action 2 repeats. Then, Dave and Gertrude swap balls. Then, Action 32 repeats. Then, Bob and Eve discuss something (let's call it Action 63). Then, Bob and Eve swap balls. Then, Claire and Alice swap balls. Then, Claire and Alice discuss something (let's call it Action 64). Then, Claire and Alice swap balls. Then, Eve and Bob swap balls. Then, Bob and Claire swap balls. Then, Claire and Bob swap balls. Then, Bob and Eve swap balls. Then, Gertrude and Bob swap balls. Then, Gertrude and Bob discuss something (let's call it Action 65). Then, Action 15 repeats. Then, nothing happens for a while. Then, Gertrude and Alice swap balls. Then, Gertrude and Alice discuss something (let's call it Action 66). Then, Gertrude and Alice swap balls. Then, Claire and Eve swap balls. Then, Action 11 repeats. Then, nothing happens for a while. Then, Fred and Alice swap balls. Then, Alice and Fred swap balls. Then, nothing happens for a while. Then, Action 49 repeats. Then, Action 28 repeats. Then, Fred and Eve swap balls. Then, Eve and Fred swap balls. Then, nothing happens for a while. Then, Bob and Claire swap balls. Then, Bob and Claire discuss something (let's call it Action 67). Then, Bob and Claire swap balls. Then, Gertrude and Bob swap balls. Then, Bob and Gertrude swap balls. Then, nothing happens for a while. Then, Gertrude and Dave swap balls. Then, Action 12 repeats. Then, Alice and Dave swap balls. Then, Dave and Gertrude swap balls. Then, Alice and Claire swap balls. Then, Eve and Fred swap balls. Finally, Eve and Dave swap balls. At the end of the game, Gertrude has the\\nOptions:\\n(A) green ball\\n(B) brown ball\\n(C) orange ball\\n(D) red ball\\n(E) blue ball\\n(F) black ball\\n(G) yellow ball\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}