{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0de9341e-3f51-5098-b981-fd9e639bbf50","task_key":"bbeh~5fshuffled~5fobjects--full--0de9341e-3f51-5098-b981-fd9e639bbf50","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are holding a white elephant…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a orange ball, Bob has a brown present, Claire has a pink ball, Dave has a blue present, Eve has a green present, Fred has a yellow present, and Gertrude has a white present.\\nAs the event progresses, pairs of people swap gifts. First, Dave and Gertrude swap their gifts. Then, Alice and Fred swap their gifts (let's call it Action 1). Then, Fred and Alice swap their gifts (let's call it Action 2). Then, Dave and Gertrude swap their gifts (let's call it Action 3). Then, Dave and Gertrude discuss something (let's call it Action 4). Then, Action 3 repeats. Then, Alice and Fred swap their gifts. Then, Fred and Alice swap their gifts. Then, Fred and Gertrude swap their gifts (let's call it Action 5). Then, Gertrude and Fred swap their gifts (let's call it Action 6). Then, nothing happens for a while (let's call it Action 7). Then, nothing happens for a while. Then, Claire and Gertrude swap their gifts (let's call it Action 8). Then, Claire and Gertrude discuss something (let's call it Action 9). Then, Action 8 repeats. Then, Action 7 repeats. Then, Bob and Gertrude swap their gifts (let's call it Action 10). Then, Gertrude and Fred swap their gifts. Then, Fred and Gertrude swap their gifts. Then, Gertrude and Bob swap their gifts (let's call it Action 11). Then, Bob and Eve swap their gifts (let's call it Action 12). Then, Bob and Eve discuss something (let's call it Action 13). Then, Bob and Eve swap their gifts. Then, Bob and Eve swap their gifts. Then, Eve and Bob swap their gifts (let's call it Action 14). Then, nothing happens for a while. Then, nothing happens for a while. Then, Eve and Claire swap their gifts (let's call it Action 15). Then, Eve and Claire discuss something (let's call it Action 16). Then, Action 15 repeats. Then, nothing happens for a while. Then, Claire and Alice swap their gifts (let's call it Action 17). Then, Claire and Alice discuss something (let's call it Action 18). Then, Claire and Alice swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Gertrude and Fred swap their gifts. Then, Action 5 repeats. Then, Action 7 repeats. Then, Gertrude and Claire swap their gifts (let's call it Action 19). Then, Claire and Gertrude swap their gifts. Then, Alice and Claire swap their gifts (let's call it Action 20). Then, Claire and Alice swap their gifts. Then, nothing happens for a while. Then, Eve and Fred swap their gifts (let's call it Action 21). Then, Fred and Eve swap their gifts (let's call it Action 22). Then, Gertrude and Claire swap their gifts. Then, Action 8 repeats. Then, Gertrude and Claire swap their gifts. Then, Claire and Bob swap their gifts (let's call it Action 23). Then, Bob and Claire swap their gifts (let's call it Action 24). Then, Claire and Gertrude swap their gifts. Then, Claire and Eve swap their gifts (let's call it Action 25). Then, Eve and Alice swap their gifts (let's call it Action 26). Then, Alice and Eve swap their gifts (let's call it Action 27). Then, Eve and Claire swap their gifts. Then, Gertrude and Eve swap their gifts (let's call it Action 28). Then, Gertrude and Eve discuss something (let's call it Action 29). Then, Action 28 repeats. Then, Claire and Fred swap their gifts (let's call it Action 30). Then, Fred and Claire swap their gifts (let's call it Action 31). Then, Gertrude and Alice swap their gifts (let's call it Action 32). Then, nothing happens for a while. Then, Gertrude and Dave swap their gifts (let's call it Action 33). Then, Gertrude and Dave discuss something (let's call it Action 34). Then, Action 33 repeats. Then, Alice and Dave swap their gifts (let's call it Action 35). Then, Dave and Fred swap their gifts (let's call it Action 36). Then, Fred and Dave swap their gifts (let's call it Action 37). Then, Dave and Alice swap their gifts (let's call it Action 38). Then, Claire and Bob swap their gifts. Then, Bob and Fred swap their gifts (let's call it Action 39). Then, Fred and Bob swap their gifts (let's call it Action 40). Then, Bob and Claire swap their gifts. Then, nothing happens for a while. Then, Bob and Eve swap their gifts. Then, Bob and Eve discuss something. Then, Bob and Eve swap their gifts. Then, Fred and Dave swap their gifts. Then, Dave and Fred swap their gifts. Then, Fred and Gertrude swap their gifts. Then, Action 32 repeats. Then, Alice and Gertrude swap their gifts (let's call it Action 41). Then, Gertrude and Fred swap their gifts. Then, Alice and Dave swap their gifts. Then, Action 38 repeats. Then, Claire and Gertrude swap their gifts. Then, Gertrude and Claire swap their gifts. Then, Alice and Dave swap their gifts. Then, Alice and Dave discuss something (let's call it Action 42). Then, Alice and Dave swap their gifts. Then, Bob and Alice swap their gifts (let's call it Action 43). Then, Alice and Bob swap their gifts (let's call it Action 44). Then, Dave and Alice swap their gifts. Then, Dave and Alice discuss something (let's call it Action 45). Then, Dave and Alice swap their gifts. Then, nothing happens for a while. Then, Eve and Alice swap their gifts. Then, Alice and Dave swap their gifts. Then, Dave and Alice swap their gifts. Then, Alice and Eve swap their gifts. Then, Action 7 repeats. Then, Dave and Bob swap their gifts (let's call it Action 46). Then, Dave and Bob discuss something (let's call it Action 47). Then, Dave and Bob swap their gifts. Then, Dave and Claire swap their gifts (let's call it Action 48). Then, Claire and Dave swap their gifts (let's call it Action 49). Then, Dave and Eve swap their gifts (let's call it Action 50). Then, Eve and Dave swap their gifts (let's call it Action 51). Then, Action 7 repeats. Then, Action 1 repeats. Then, Alice and Fred discuss something (let's call it Action 52). Then, Alice and Fred swap their gifts. Then, Fred and Dave swap their gifts. Then, Fred and Dave discuss something (let's call it Action 53). Then, Action 37 repeats. Then, Gertrude and Dave swap their gifts. Then, Action 34 repeats. Then, Action 33 repeats. Then, Bob and Claire swap their gifts. Then, Bob and Claire discuss something (let's call it Action 54). Then, Bob and Claire swap their gifts. Then, nothing happens for a while. Then, Fred and Bob swap their gifts. Then, Action 39 repeats. Then, Bob and Fred swap their gifts. Then, Bob and Fred discuss something (let's call it Action 55). Then, Bob and Fred swap their gifts. Then, nothing happens for a while. Then, Alice and Fred swap their gifts. Then, Action 2 repeats. Then, Fred and Alice swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Action 32 repeats. Then, Alice and Fred swap their gifts. Then, Eve and Fred swap their gifts. Then, Action 22 repeats. Then, Claire and Eve swap their gifts. Then, Claire and Eve discuss something (let's call it Action 56). Then, Claire and Eve swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, Eve and Gertrude swap their gifts (let's call it Action 57). Then, Action 11 repeats. Then, Bob and Gertrude swap their gifts. Then, Gertrude and Eve swap their gifts. Then, Alice and Dave swap their gifts. Then, Dave and Alice swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Gertrude and Alice discuss something (let's call it Action 58). Then, Action 32 repeats. Then, Alice and Dave swap their gifts. Then, Dave and Alice swap their gifts. Then, Bob and Gertrude swap their gifts. Then, Gertrude and Bob swap their gifts. Then, nothing happens for a while. Then, Claire and Bob swap their gifts. Then, Claire and Dave swap their gifts. Then, Claire and Dave discuss something (let's call it Action 59). Then, Claire and Dave swap their gifts. Then, Action 7 repeats. Then, Gertrude and Dave swap their gifts. Then, Action 34 repeats. Then, Gertrude and Dave swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Alice and Gertrude discuss something (let's call it Action 60). Then, Action 41 repeats. Then, nothing happens for a while. Then, Claire and Alice swap their gifts. Then, Action 18 repeats. Then, Claire and Alice swap their gifts. Then, Gertrude and Eve swap their gifts. Then, Action 29 repeats. Then, Gertrude and Eve swap their gifts. Then, Gertrude and Dave swap their gifts. Then, Dave and Gertrude swap their gifts. Then, Dave and Bob swap their gifts. Then, Bob and Claire swap their gifts. Then, Claire and Bob swap their gifts. Then, Bob and Dave swap their gifts (let's call it Action 61). Then, Action 30 repeats. Then, Fred and Claire swap their gifts. Then, Action 30 repeats. Then, Claire and Fred discuss something (let's call it Action 62). Then, Claire and Fred swap their gifts. Then, Bob and Fred swap their gifts. Then, Bob and Fred discuss something. Then, Bob and Fred swap their gifts. Then, Fred and Alice swap their gifts. Then, Action 44 repeats. Then, Bob and Alice swap their gifts. Then, Action 1 repeats. Then, Alice and Eve swap their gifts. Then, Alice and Eve discuss something (let's call it Action 63). Then, Alice and Eve swap their gifts. Then, nothing happens for a while. Then, Bob and Fred swap their gifts. Then, Fred and Bob swap their gifts. Then, Action 2 repeats. Then, Alice and Fred swap their gifts. Then, Action 49 repeats. Then, Dave and Eve swap their gifts. Then, Action 51 repeats. Then, Dave and Claire swap their gifts. Then, nothing happens for a while. Then, Dave and Alice swap their gifts. Then, Dave and Alice discuss something. Then, Action 38 repeats. Then, nothing happens for a while. Then, Action 49 repeats. Then, Dave and Fred swap their gifts. Then, Action 37 repeats. Then, Action 48 repeats. Then, Alice and Gertrude swap their gifts. Then, Action 28 repeats. Then, Eve and Gertrude swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Eve and Claire swap their gifts. Then, Eve and Claire swap their gifts. Then, Eve and Claire discuss something. Then, Action 15 repeats. Then, Fred and Gertrude swap their gifts. Then, Fred and Gertrude discuss something (let's call it Action 64). Then, Fred and Gertrude swap their gifts. Then, nothing happens for a while. Then, Bob and Fred swap their gifts. Then, Action 37 repeats. Then, Action 36 repeats. Then, Fred and Bob swap their gifts. Then, Claire and Fred swap their gifts. Then, Fred and Alice swap their gifts. Then, Alice and Fred swap their gifts. Then, Fred and Claire swap their gifts. Then, nothing happens for a while. Then, Eve and Gertrude swap their gifts. Then, Eve and Gertrude discuss something (let's call it Action 65). Then, Action 57 repeats. Then, Bob and Gertrude swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Action 11 repeats. Then, Bob and Eve swap their gifts. Then, Bob and Eve discuss something. Then, Bob and Eve swap their gifts. Then, Bob and Dave swap their gifts. Then, Bob and Dave discuss something (let's call it Action 66). Then, Bob and Dave swap their gifts. Then, Eve and Gertrude swap their gifts. Then, Gertrude and Fred swap their gifts. Then, Fred and Gertrude swap their gifts. Then, Action 28 repeats. Then, Action 7 repeats. Then, Bob and Claire swap their gifts. Then, Bob and Claire discuss something. Then, Action 24 repeats. Then, nothing happens for a while. Then, Action 28 repeats. Then, Action 15 repeats. Then, Claire and Eve swap their gifts. Then, Action 57 repeats. Then, nothing happens for a while. Then, Gertrude and Eve swap their gifts. Then, Eve and Gertrude swap their gifts. Then, Gertrude and Bob swap their gifts. Then, Action 24 repeats. Then, Claire and Bob swap their gifts. Then, Bob and Gertrude swap their gifts. Then, Action 11 repeats. Then, Action 61 repeats. Then, Dave and Bob swap their gifts. Then, Bob and Gertrude swap their gifts. Then, Fred and Dave swap their gifts. Then, Dave and Bob swap their gifts. Then, Bob and Dave swap their gifts. Then, Dave and Fred swap their gifts. Then, Eve and Bob swap their gifts. Then, Action 12 repeats. Then, Claire and Dave swap their gifts. Then, Action 59 repeats. Then, Claire and Dave swap their gifts. Then, Action 39 repeats. Then, Action 2 repeats. Then, Alice and Fred swap their gifts. Then, Fred and Bob swap their gifts. Then, Action 23 repeats. Then, Claire and Bob discuss something (let's call it Action 67). Then, Claire and Bob swap their gifts. Then, Action 30 repeats. Then, Fred and Eve swap their gifts. Then, Action 21 repeats. Then, Fred and Claire swap their gifts. Then, Eve and Fred swap their gifts. Then, Eve and Fred discuss something (let's call it Action 68). Then, Eve and Fred swap their gifts. Then, Action 7 repeats. Then, Action 37 repeats. Then, Action 38 repeats. Then, Alice and Dave swap their gifts. Then, Dave and Fred swap their gifts. Then, Bob and Fred swap their gifts. Then, Action 2 repeats. Then, Action 1 repeats. Then, Fred and Bob swap their gifts. Then, Dave and Fred swap their gifts. Then, Fred and Gertrude swap their gifts. Then, Action 6 repeats. Then, Action 37 repeats. Then, Fred and Alice swap their gifts. Then, Gertrude and Alice swap their gifts. Finally, Bob and Gertrude swap their gifts. At the end of the event, Alice has the\\nOptions:\\n(A) orange ball\\n(B) brown present\\n(C) pink ball\\n(D) blue present\\n(E) green present\\n(F) yellow present\\n(G) white present\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}