{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ae35e81f-f898-5af9-ae7c-a5d3f1a16895","task_key":"bbeh~5fshuffled~5fobjects--full--ae35e81f-f898-5af9-ae7c-a5d3f1a16895","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are holding a white elephant…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a black ball, Bob has a orange ball, Claire has a white present, Dave has a green present, Eve has a blue present, Fred has a red present, and Gertrude has a yellow present.\\nAs the event progresses, pairs of people swap gifts. First, Alice and Eve swap their gifts. Then, Bob and Claire swap their gifts (let's call it Action 1). Then, Bob and Claire discuss something (let's call it Action 2). Then, Action 1 repeats. Then, nothing happens for a while (let's call it Action 3). Then, Alice and Dave swap their gifts (let's call it Action 4). Then, Alice and Dave discuss something (let's call it Action 5). Then, Alice and Dave swap their gifts. Then, nothing happens for a while. Then, Action 3 repeats. Then, Eve and Fred swap their gifts (let's call it Action 6). Then, Eve and Fred discuss something (let's call it Action 7). Then, Eve and Fred swap their gifts. Then, nothing happens for a while. Then, Fred and Bob swap their gifts (let's call it Action 8). Then, Fred and Bob discuss something (let's call it Action 9). Then, Action 8 repeats. Then, Alice and Fred swap their gifts (let's call it Action 10). Then, Alice and Fred discuss something (let's call it Action 11). Then, Alice and Fred swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Eve swap their gifts (let's call it Action 12). Then, Eve and Alice swap their gifts (let's call it Action 13). Then, Alice and Eve swap their gifts (let's call it Action 14). Then, Eve and Dave swap their gifts (let's call it Action 15). Then, Gertrude and Claire swap their gifts (let's call it Action 16). Then, Claire and Gertrude swap their gifts (let's call it Action 17). Then, Dave and Claire swap their gifts (let's call it Action 18). Then, Claire and Eve swap their gifts (let's call it Action 19). Then, Eve and Claire swap their gifts (let's call it Action 20). Then, Claire and Dave swap their gifts (let's call it Action 21). Then, Gertrude and Eve swap their gifts (let's call it Action 22). Then, Gertrude and Eve discuss something (let's call it Action 23). Then, Gertrude and Eve swap their gifts. Then, Action 6 repeats. Then, Fred and Eve swap their gifts (let's call it Action 24). Then, nothing happens for a while. Then, Dave and Eve swap their gifts. Then, Eve and Dave swap their gifts. Then, Action 15 repeats. Then, Dave and Fred swap their gifts (let's call it Action 25). Then, Fred and Dave swap their gifts (let's call it Action 26). Then, Dave and Eve swap their gifts. Then, Claire and Eve swap their gifts. Then, Claire and Eve discuss something (let's call it Action 27). Then, Claire and Eve swap their gifts. Then, Gertrude and Dave swap their gifts (let's call it Action 28). Then, Dave and Claire swap their gifts. Then, Action 21 repeats. Then, Dave and Gertrude swap their gifts (let's call it Action 29). Then, Action 10 repeats. Then, Alice and Fred discuss something. Then, Alice and Fred swap their gifts. Then, Alice and Gertrude swap their gifts (let's call it Action 30). Then, Alice and Gertrude discuss something (let's call it Action 31). Then, Alice and Gertrude swap their gifts. Then, Dave and Gertrude swap their gifts. Then, Dave and Gertrude discuss something (let's call it Action 32). Then, Dave and Gertrude swap their gifts. Then, Eve and Claire swap their gifts. Then, Claire and Dave swap their gifts. Then, Dave and Claire swap their gifts. Then, Action 19 repeats. Then, nothing happens for a while. Then, Claire and Bob swap their gifts (let's call it Action 33). Then, Claire and Bob discuss something (let's call it Action 34). Then, Claire and Bob swap their gifts. Then, Fred and Claire swap their gifts (let's call it Action 35). Then, Claire and Fred swap their gifts (let's call it Action 36). Then, Dave and Claire swap their gifts. Then, Dave and Claire discuss something (let's call it Action 37). Then, Dave and Claire swap their gifts. Then, nothing happens for a while. Then, Alice and Bob swap their gifts (let's call it Action 38). Then, Bob and Alice swap their gifts (let's call it Action 39). Then, Dave and Eve swap their gifts. Then, Eve and Dave swap their gifts. Then, Fred and Alice swap their gifts (let's call it Action 40). Then, Fred and Alice discuss something (let's call it Action 41). Then, Fred and Alice swap their gifts. Then, Dave and Claire swap their gifts. Then, Action 21 repeats. Then, Gertrude and Fred swap their gifts (let's call it Action 42). Then, Fred and Gertrude swap their gifts (let's call it Action 43). Then, nothing happens for a while. Then, Action 12 repeats. Then, Dave and Eve discuss something (let's call it Action 44). Then, Dave and Eve swap their gifts. Then, Fred and Dave swap their gifts. Then, Action 18 repeats. Then, Claire and Dave swap their gifts. Then, Action 25 repeats. Then, Eve and Gertrude swap their gifts (let's call it Action 45). Then, Gertrude and Eve swap their gifts. Then, Dave and Alice swap their gifts (let's call it Action 46). Then, Action 38 repeats. Then, Bob and Alice swap their gifts. Then, Alice and Dave swap their gifts. Then, nothing happens for a while. Then, Eve and Alice swap their gifts. Then, Eve and Alice discuss something (let's call it Action 47). Then, Eve and Alice swap their gifts. Then, nothing happens for a while. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Claire swap their gifts. Then, Claire and Gertrude swap their gifts. Then, Gertrude and Alice swap their gifts (let's call it Action 48). Then, Alice and Fred swap their gifts. Then, Bob and Dave swap their gifts (let's call it Action 49). Then, Bob and Dave discuss something (let's call it Action 50). Then, Bob and Dave swap their gifts. Then, Fred and Claire swap their gifts. Then, Claire and Fred swap their gifts. Then, Action 3 repeats. Then, nothing happens for a while. Then, Bob and Alice swap their gifts. Then, Alice and Bob swap their gifts. Then, Dave and Eve swap their gifts. Then, Eve and Dave swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, Action 28 repeats. Then, Dave and Gertrude swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Fred and Alice swap their gifts. Then, Alice and Dave swap their gifts. Then, Dave and Alice swap their gifts. Then, Alice and Fred swap their gifts. Then, nothing happens for a while. Then, Gertrude and Bob swap their gifts (let's call it Action 51). Then, Bob and Gertrude swap their gifts (let's call it Action 52). Then, Action 14 repeats. Then, Eve and Bob swap their gifts (let's call it Action 53). Then, Bob and Eve swap their gifts (let's call it Action 54). Then, Eve and Alice swap their gifts. Then, Fred and Alice swap their gifts. Then, Alice and Fred swap their gifts. Then, Bob and Dave swap their gifts. Then, Action 18 repeats. Then, Claire and Dave swap their gifts. Then, Dave and Bob swap their gifts (let's call it Action 55). Then, nothing happens for a while. Then, Alice and Bob swap their gifts. Then, Bob and Alice swap their gifts. Then, Gertrude and Dave swap their gifts. Then, Action 28 repeats. Then, Gertrude and Dave discuss something (let's call it Action 56). Then, Gertrude and Dave swap their gifts. Then, Dave and Eve swap their gifts. Then, Action 15 repeats. Then, Claire and Eve swap their gifts. Then, Eve and Claire swap their gifts. Then, nothing happens for a while. Then, Fred and Claire swap their gifts. Then, Fred and Claire discuss something (let's call it Action 57). Then, Fred and Claire swap their gifts. Then, nothing happens for a while. Then, Bob and Fred swap their gifts (let's call it Action 58). Then, Bob and Fred discuss something (let's call it Action 59). Then, Bob and Fred swap their gifts. Then, nothing happens for a while. Then, Action 3 repeats. Then, Action 16 repeats. Then, Claire and Gertrude swap their gifts. Then, Eve and Dave swap their gifts. Then, Eve and Dave discuss something (let's call it Action 60). Then, Eve and Dave swap their gifts. Then, Gertrude and Bob swap their gifts. Then, Gertrude and Bob discuss something (let's call it Action 61). Then, Gertrude and Bob swap their gifts. Then, Claire and Eve swap their gifts. Then, Eve and Fred swap their gifts. Then, Fred and Eve swap their gifts. Then, Eve and Claire swap their gifts. Then, Action 3 repeats. Then, Gertrude and Dave swap their gifts. Then, Gertrude and Dave discuss something. Then, Action 28 repeats. Then, Claire and Eve swap their gifts. Then, Claire and Eve discuss something. Then, Claire and Eve swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, Bob and Alice swap their gifts. Then, Bob and Alice discuss something (let's call it Action 62). Then, Action 39 repeats. Then, Gertrude and Dave swap their gifts. Then, Action 55 repeats. Then, Action 49 repeats. Then, Dave and Gertrude swap their gifts. Then, Alice and Bob swap their gifts. Then, Alice and Bob discuss something (let's call it Action 63). Then, Alice and Bob swap their gifts. Then, Action 38 repeats. Then, Action 1 repeats. Then, Claire and Bob swap their gifts. Then, Bob and Alice swap their gifts. Then, Action 3 repeats. Then, nothing happens for a while. Then, Claire and Eve swap their gifts. Then, Claire and Eve discuss something. Then, Claire and Eve swap their gifts. Then, Action 19 repeats. Then, Claire and Eve discuss something. Then, Claire and Eve swap their gifts. Then, Gertrude and Bob swap their gifts. Then, Bob and Gertrude swap their gifts. Then, Action 16 repeats. Then, Action 17 repeats. Then, nothing happens for a while. Then, Claire and Fred swap their gifts. Then, Claire and Fred discuss something (let's call it Action 64). Then, Action 36 repeats. Then, Gertrude and Alice swap their gifts. Then, Alice and Claire swap their gifts (let's call it Action 65). Then, Claire and Alice swap their gifts (let's call it Action 66). Then, Alice and Gertrude swap their gifts. Then, Fred and Eve swap their gifts. Then, Fred and Claire swap their gifts. Then, Fred and Claire discuss something. Then, Fred and Claire swap their gifts. Then, Claire and Eve swap their gifts. Then, Eve and Claire swap their gifts. Then, Claire and Alice swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Alice and Claire swap their gifts. Then, nothing happens for a while. Then, Claire and Alice swap their gifts. Then, Alice and Fred swap their gifts. Then, Fred and Alice swap their gifts. Then, Action 65 repeats. Then, Action 58 repeats. Then, Fred and Alice swap their gifts. Then, Alice and Fred swap their gifts. Then, Action 8 repeats. Then, Fred and Eve swap their gifts. Then, Fred and Eve discuss something (let's call it Action 67). Then, Action 24 repeats. Then, nothing happens for a while. Then, Claire and Gertrude swap their gifts. Then, Claire and Gertrude discuss something (let's call it Action 68). Then, Claire and Gertrude swap their gifts. Then, Fred and Gertrude swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Fred swap their gifts. Then, Bob and Gertrude swap their gifts. Then, Bob and Gertrude discuss something (let's call it Action 69). Then, Bob and Gertrude swap their gifts. Then, Action 15 repeats. Then, Dave and Alice swap their gifts. Then, Action 4 repeats. Then, Dave and Eve swap their gifts. Then, Action 3 repeats. Then, Action 29 repeats. Then, Gertrude and Dave swap their gifts. Then, nothing happens for a while. Then, Eve and Alice swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Action 48 repeats. Then, Alice and Eve swap their gifts. Then, Action 1 repeats. Then, Claire and Alice swap their gifts. Then, Action 65 repeats. Then, Action 33 repeats. Then, Action 18 repeats. Then, Claire and Eve swap their gifts. Then, Eve and Claire swap their gifts. Then, Claire and Dave swap their gifts. Then, Action 8 repeats. Then, Action 9 repeats. Then, Action 8 repeats. Then, nothing happens for a while. Then, Claire and Gertrude swap their gifts. Then, Gertrude and Bob swap their gifts. Then, Action 52 repeats. Then, Action 16 repeats. Then, Dave and Claire swap their gifts. Then, Action 21 repeats. Then, Action 36 repeats. Then, Fred and Claire swap their gifts. Then, nothing happens for a while. Then, Alice and Claire swap their gifts. Then, Claire and Eve swap their gifts. Then, Action 20 repeats. Then, Claire and Alice swap their gifts. Then, Action 6 repeats. Then, Fred and Bob swap their gifts. Then, Action 58 repeats. Then, Fred and Eve swap their gifts. Then, Alice and Bob swap their gifts. Then, Bob and Alice swap their gifts. Then, Dave and Fred swap their gifts. Then, Action 26 repeats. Then, Action 16 repeats. Then, Gertrude and Claire discuss something (let's call it Action 70). Then, Gertrude and Claire swap their gifts. Then, Bob and Claire swap their gifts. Then, Claire and Dave swap their gifts. Then, Dave and Claire swap their gifts. Then, Action 33 repeats. Then, Action 45 repeats. Then, Eve and Gertrude discuss something (let's call it Action 71). Then, Eve and Gertrude swap their gifts. Then, Eve and Dave swap their gifts. Then, Action 12 repeats. Then, Gertrude and Fred swap their gifts. Then, Fred and Alice swap their gifts. Then, Alice and Fred swap their gifts. Then, Fred and Gertrude swap their gifts. Then, Eve and Fred swap their gifts. Then, Action 7 repeats. Then, Eve and Fred swap their gifts. Then, Claire and Bob swap their gifts. Then, Claire and Dave swap their gifts. Finally, Dave and Alice swap their gifts. At the end of the event, Alice has the\\nOptions:\\n(A) black ball\\n(B) orange ball\\n(C) white present\\n(D) green present\\n(E) blue present\\n(F) red present\\n(G) yellow present\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}