{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"aa268087-ad05-5979-ae6a-343a3df7a1e8","task_key":"bbeh~5fshuffled~5fobjects--full--aa268087-ad05-5979-ae6a-343a3df7a1e8","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are holding a white elephant…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a pink ball, Bob has a brown present, Claire has a red present, Dave has a blue present, Eve has a white present, Fred has a orange ball, and Gertrude has a black ball.\\nAs the event progresses, pairs of people swap gifts. First, Alice and Dave swap their gifts. Then, Fred and Claire swap their gifts (let's call it Action 1). Then, Fred and Claire discuss something (let's call it Action 2). Then, Action 1 repeats. Then, Alice and Bob swap their gifts (let's call it Action 3). Then, Alice and Bob discuss something (let's call it Action 4). Then, Action 3 repeats. Then, Gertrude and Bob swap their gifts (let's call it Action 5). Then, Bob and Alice swap their gifts (let's call it Action 6). Then, Alice and Bob swap their gifts. Then, Bob and Gertrude swap their gifts (let's call it Action 7). Then, Bob and Claire swap their gifts (let's call it Action 8). Then, Bob and Claire discuss something (let's call it Action 9). Then, Bob and Claire swap their gifts. Then, nothing happens for a while (let's call it Action 10). Then, Claire and Dave swap their gifts (let's call it Action 11). Then, Dave and Fred swap their gifts (let's call it Action 12). Then, Fred and Dave swap their gifts (let's call it Action 13). Then, Dave and Claire swap their gifts (let's call it Action 14). Then, nothing happens for a while. Then, Eve and Claire swap their gifts (let's call it Action 15). Then, Claire and Gertrude swap their gifts (let's call it Action 16). Then, Gertrude and Claire swap their gifts (let's call it Action 17). Then, Claire and Eve swap their gifts (let's call it Action 18). Then, Fred and Alice swap their gifts (let's call it Action 19). Then, Alice and Fred swap their gifts (let's call it Action 20). Then, Eve and Fred swap their gifts (let's call it Action 21). Then, Fred and Eve swap their gifts (let's call it Action 22). Then, Dave and Bob swap their gifts (let's call it Action 23). Then, Bob and Gertrude swap their gifts. Then, Gertrude and Bob swap their gifts. Then, Bob and Dave swap their gifts (let's call it Action 24). Then, Action 15 repeats. Then, Claire and Gertrude swap their gifts. Then, Gertrude and Claire swap their gifts. Then, Claire and Eve swap their gifts. Then, Action 10 repeats. Then, Alice and Bob swap their gifts. Then, Bob and Alice swap their gifts. Then, nothing happens for a while. Then, Alice and Gertrude swap their gifts (let's call it Action 25). Then, Alice and Gertrude discuss something (let's call it Action 26). Then, Alice and Gertrude swap their gifts. Then, Claire and Alice swap their gifts (let's call it Action 27). Then, Alice and Claire swap their gifts (let's call it Action 28). Then, Action 10 repeats. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Alice swap their gifts (let's call it Action 29). Then, Eve and Bob swap their gifts (let's call it Action 30). Then, Bob and Fred swap their gifts (let's call it Action 31). Then, Fred and Bob swap their gifts (let's call it Action 32). Then, Bob and Eve swap their gifts (let's call it Action 33). Then, Claire and Gertrude swap their gifts. Then, Action 5 repeats. Then, Bob and Gertrude swap their gifts. Then, Action 17 repeats. Then, Action 16 repeats. Then, Action 17 repeats. Then, Alice and Dave swap their gifts (let's call it Action 34). Then, Dave and Eve swap their gifts (let's call it Action 35). Then, Eve and Dave swap their gifts (let's call it Action 36). Then, Dave and Alice swap their gifts (let's call it Action 37). Then, nothing happens for a while. Then, Fred and Bob swap their gifts. Then, Action 31 repeats. Then, Dave and Bob swap their gifts. Then, Bob and Gertrude swap their gifts. Then, Action 5 repeats. Then, Action 24 repeats. Then, nothing happens for a while. Then, Eve and Fred swap their gifts. Then, Fred and Eve swap their gifts. Then, nothing happens for a while. Then, Claire and Dave swap their gifts. Then, Bob and Eve swap their gifts. Then, Action 30 repeats. Then, Gertrude and Fred swap their gifts (let's call it Action 38). Then, Fred and Dave swap their gifts. Then, Action 12 repeats. Then, Fred and Gertrude swap their gifts (let's call it Action 39). Then, Action 27 repeats. Then, Claire and Alice discuss something (let's call it Action 40). Then, Action 27 repeats. Then, Fred and Alice swap their gifts. Then, Fred and Alice discuss something (let's call it Action 41). Then, Fred and Alice swap their gifts. Then, Eve and Alice swap their gifts (let's call it Action 42). Then, Alice and Fred swap their gifts. Then, Fred and Alice swap their gifts. Then, Alice and Eve swap their gifts (let's call it Action 43). Then, Action 8 repeats. Then, Bob and Claire discuss something. Then, Bob and Claire swap their gifts. Then, Gertrude and Dave swap their gifts (let's call it Action 44). Then, Gertrude and Dave discuss something (let's call it Action 45). Then, Gertrude and Dave swap their gifts. Then, nothing happens for a while. Then, Gertrude and Claire swap their gifts. Then, Claire and Gertrude swap their gifts. Then, nothing happens for a while. Then, Action 10 repeats. Then, Eve and Dave swap their gifts. Then, Dave and Eve swap their gifts. Then, nothing happens for a while. Then, Fred and Eve swap their gifts. Then, Fred and Eve discuss something (let's call it Action 46). Then, Fred and Eve swap their gifts. Then, Eve and Dave swap their gifts. Then, Eve and Dave discuss something (let's call it Action 47). Then, Eve and Dave swap their gifts. Then, Gertrude and Eve swap their gifts (let's call it Action 48). Then, Gertrude and Eve discuss something (let's call it Action 49). Then, Action 48 repeats. Then, Alice and Claire swap their gifts. Then, Alice and Claire discuss something (let's call it Action 50). Then, Alice and Claire swap their gifts. Then, Action 33 repeats. Then, Bob and Eve discuss something (let's call it Action 51). Then, Action 33 repeats. Then, Action 10 repeats. Then, Fred and Claire swap their gifts. Then, Claire and Bob swap their gifts (let's call it Action 52). Then, Action 8 repeats. Then, Claire and Fred swap their gifts (let's call it Action 53). Then, Fred and Eve swap their gifts. Then, Eve and Fred swap their gifts. Then, Eve and Gertrude swap their gifts (let's call it Action 54). Then, Gertrude and Eve swap their gifts. Then, Action 10 repeats. Then, Claire and Dave swap their gifts. Then, Action 12 repeats. Then, Fred and Dave swap their gifts. Then, Dave and Claire swap their gifts. Then, nothing happens for a while. Then, Eve and Dave swap their gifts. Then, Dave and Claire swap their gifts. Then, Action 11 repeats. Then, Dave and Eve swap their gifts. Then, Action 5 repeats. Then, Bob and Alice swap their gifts. Then, Action 3 repeats. Then, Action 7 repeats. Then, Action 7 repeats. Then, Bob and Gertrude discuss something (let's call it Action 55). Then, Action 7 repeats. Then, Dave and Alice swap their gifts. Then, Action 28 repeats. Then, Claire and Alice swap their gifts. Then, Action 34 repeats. Then, Eve and Dave swap their gifts. Then, Action 35 repeats. Then, Bob and Gertrude swap their gifts. Then, Gertrude and Bob swap their gifts. Then, nothing happens for a while. Then, Fred and Gertrude swap their gifts. Then, Action 20 repeats. Then, Fred and Alice swap their gifts. Then, Action 19 repeats. Then, Alice and Fred swap their gifts. Then, nothing happens for a while. Then, Dave and Bob swap their gifts. Then, Bob and Dave swap their gifts. Then, Dave and Eve swap their gifts. Then, Dave and Eve discuss something (let's call it Action 56). Then, Action 35 repeats. Then, Bob and Alice swap their gifts. Then, Bob and Alice discuss something (let's call it Action 57). Then, Bob and Alice swap their gifts. Then, Action 21 repeats. Then, Action 13 repeats. Then, Action 12 repeats. Then, Fred and Eve swap their gifts. Then, Action 13 repeats. Then, Dave and Gertrude swap their gifts (let's call it Action 58). Then, Gertrude and Dave swap their gifts. Then, Dave and Fred swap their gifts. Then, Action 1 repeats. Then, Fred and Claire discuss something. Then, Fred and Claire swap their gifts. Then, Action 21 repeats. Then, Action 19 repeats. Then, Action 20 repeats. Then, Fred and Eve swap their gifts. Then, Fred and Claire swap their gifts. Then, Claire and Fred swap their gifts. Then, Gertrude and Dave swap their gifts. Then, Action 45 repeats. Then, Gertrude and Dave swap their gifts. Then, Action 38 repeats. Then, Gertrude and Fred discuss something (let's call it Action 59). Then, Action 38 repeats. Then, Gertrude and Fred swap their gifts. Then, Fred and Gertrude swap their gifts. Then, nothing happens for a while. Then, Action 10 repeats. Then, Action 31 repeats. Then, Fred and Alice swap their gifts. Then, Action 20 repeats. Then, Fred and Bob swap their gifts. Then, Eve and Alice swap their gifts. Then, Alice and Eve swap their gifts. Then, Dave and Claire swap their gifts. Then, Dave and Claire discuss something (let's call it Action 60). Then, Action 14 repeats. Then, Claire and Eve swap their gifts. Then, Action 15 repeats. Then, Bob and Claire swap their gifts. Then, Action 27 repeats. Then, Alice and Claire swap their gifts. Then, Claire and Bob swap their gifts. Then, nothing happens for a while. Then, Action 30 repeats. Then, Bob and Eve swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Claire swap their gifts. Then, Action 50 repeats. Then, Action 28 repeats. Then, Action 23 repeats. Then, Bob and Dave swap their gifts. Then, Eve and Gertrude swap their gifts. Then, Eve and Bob swap their gifts. Then, Eve and Bob discuss something (let's call it Action 61). Then, Eve and Bob swap their gifts. Then, Dave and Alice swap their gifts. Then, Dave and Alice discuss something (let's call it Action 62). Then, Action 37 repeats. Then, Action 54 repeats. Then, Eve and Gertrude discuss something (let's call it Action 63). Then, Eve and Gertrude swap their gifts. Then, Action 43 repeats. Then, Alice and Eve discuss something (let's call it Action 64). Then, Alice and Eve swap their gifts. Then, Eve and Alice swap their gifts. Then, Alice and Eve swap their gifts. Then, Action 10 repeats. Then, Action 10 repeats. Then, Fred and Gertrude swap their gifts. Then, Fred and Gertrude discuss something (let's call it Action 65). Then, Fred and Gertrude swap their gifts. Then, Action 34 repeats. Then, Alice and Dave discuss something (let's call it Action 66). Then, Alice and Dave swap their gifts. Then, Action 10 repeats. Then, Action 7 repeats. Then, Bob and Gertrude discuss something. Then, Action 7 repeats. Then, Bob and Dave swap their gifts. Then, Bob and Dave discuss something (let's call it Action 67). Then, Action 24 repeats. Then, Fred and Gertrude swap their gifts. Then, Action 65 repeats. Then, Action 39 repeats. Then, Bob and Claire swap their gifts. Then, Claire and Alice swap their gifts. Then, Alice and Claire swap their gifts. Then, Claire and Bob swap their gifts. Then, Gertrude and Fred swap their gifts. Then, Action 59 repeats. Then, Gertrude and Fred swap their gifts. Then, Action 3 repeats. Then, Alice and Bob discuss something. Then, Alice and Bob swap their gifts. Then, Dave and Bob swap their gifts. Then, Bob and Eve swap their gifts. Then, Eve and Bob swap their gifts. Then, Bob and Dave swap their gifts. Then, Eve and Alice swap their gifts. Then, Alice and Eve swap their gifts. Then, Action 10 repeats. Then, nothing happens for a while. Then, nothing happens for a while. Then, Fred and Eve swap their gifts. Then, Action 46 repeats. Then, Fred and Eve swap their gifts. Then, Gertrude and Claire swap their gifts. Then, Claire and Fred swap their gifts. Then, Fred and Claire swap their gifts. Then, Action 16 repeats. Then, nothing happens for a while. Then, Eve and Gertrude swap their gifts. Then, Gertrude and Eve swap their gifts. Then, Action 16 repeats. Then, Gertrude and Claire swap their gifts. Then, Eve and Alice swap their gifts. Then, Dave and Bob swap their gifts. Finally, Gertrude and Dave swap their gifts. At the end of the event, Bob has the\\nOptions:\\n(A) pink ball\\n(B) brown present\\n(C) red present\\n(D) blue present\\n(E) white present\\n(F) orange ball\\n(G) black ball\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}