{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"787f907e-d1ea-58a6-be6b-9947eea74659","task_key":"bbeh~5fshuffled~5fobjects--full--787f907e-d1ea-58a6-be6b-9947eea74659","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are holding a white elephant…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a blue present, Bob has a green present, Claire has a brown present, Dave has a purple present, Eve has a orange ball, Fred has a pink ball, and Gertrude has a red present.\\nAs the event progresses, pairs of people swap gifts. First, Eve and Claire swap their gifts. Then, Eve and Gertrude swap their gifts (let's call it Action 1). Then, Gertrude and Eve swap their gifts (let's call it Action 2). Then, Dave and Gertrude swap their gifts (let's call it Action 3). Then, Gertrude and Alice swap their gifts (let's call it Action 4). Then, Alice and Gertrude swap their gifts (let's call it Action 5). Then, Gertrude and Dave swap their gifts (let's call it Action 6). Then, Gertrude and Fred swap their gifts (let's call it Action 7). Then, Fred and Gertrude swap their gifts (let's call it Action 8). Then, nothing happens for a while (let's call it Action 9). Then, Alice and Dave swap their gifts (let's call it Action 10). Then, Dave and Alice swap their gifts (let's call it Action 11). Then, Gertrude and Dave swap their gifts. Then, Gertrude and Dave discuss something (let's call it Action 12). Then, Gertrude and Dave swap their gifts. Then, Alice and Dave swap their gifts. Then, Alice and Dave discuss something (let's call it Action 13). Then, Alice and Dave swap their gifts. Then, nothing happens for a while. Then, Eve and Dave swap their gifts (let's call it Action 14). Then, Dave and Eve swap their gifts (let's call it Action 15). Then, Action 15 repeats. Then, Eve and Dave swap their gifts. Then, Alice and Bob swap their gifts (let's call it Action 16). Then, Bob and Eve swap their gifts (let's call it Action 17). Then, Eve and Bob swap their gifts (let's call it Action 18). Then, Bob and Alice swap their gifts (let's call it Action 19). Then, Alice and Claire swap their gifts (let's call it Action 20). Then, Claire and Alice swap their gifts (let's call it Action 21). Then, nothing happens for a while. Then, Action 9 repeats. Then, Bob and Alice swap their gifts. Then, Alice and Dave swap their gifts. Then, Dave and Alice swap their gifts. Then, Alice and Bob swap their gifts. Then, Action 5 repeats. Then, Gertrude and Bob swap their gifts (let's call it Action 22). Then, Bob and Gertrude swap their gifts (let's call it Action 23). Then, Gertrude and Alice swap their gifts. Then, Fred and Eve swap their gifts (let's call it Action 24). Then, Eve and Bob swap their gifts. Then, Action 17 repeats. Then, Eve and Fred swap their gifts (let's call it Action 25). Then, nothing happens for a while. Then, nothing happens for a while. Then, Gertrude and Dave swap their gifts. Then, Dave and Gertrude swap their gifts. Then, nothing happens for a while. Then, Action 9 repeats. Then, Bob and Eve swap their gifts. Then, Bob and Eve discuss something (let's call it Action 26). Then, Bob and Eve swap their gifts. Then, Gertrude and Fred swap their gifts. Then, Fred and Dave swap their gifts (let's call it Action 27). Then, Dave and Fred swap their gifts (let's call it Action 28). Then, Fred and Gertrude swap their gifts. Then, Claire and Dave swap their gifts (let's call it Action 29). Then, Claire and Dave discuss something (let's call it Action 30). Then, Claire and Dave swap their gifts. Then, nothing happens for a while. Then, Fred and Dave swap their gifts. Then, Dave and Bob swap their gifts (let's call it Action 31). Then, Bob and Dave swap their gifts (let's call it Action 32). Then, Action 28 repeats. Then, Action 9 repeats. Then, Action 4 repeats. Then, Gertrude and Alice discuss something (let's call it Action 33). Then, Action 4 repeats. Then, Bob and Claire swap their gifts (let's call it Action 34). Then, Bob and Claire discuss something (let's call it Action 35). Then, Bob and Claire swap their gifts. Then, Fred and Alice swap their gifts (let's call it Action 36). Then, Fred and Alice discuss something (let's call it Action 37). Then, Action 36 repeats. Then, Eve and Claire swap their gifts (let's call it Action 38). Then, Eve and Claire discuss something (let's call it Action 39). Then, Eve and Claire swap their gifts. Then, Dave and Fred swap their gifts. Then, Dave and Fred discuss something (let's call it Action 40). Then, Action 28 repeats. Then, Bob and Claire swap their gifts. Then, Claire and Gertrude swap their gifts (let's call it Action 41). Then, Gertrude and Claire swap their gifts (let's call it Action 42). Then, Claire and Bob swap their gifts (let's call it Action 43). Then, Bob and Eve swap their gifts. Then, Action 18 repeats. Then, Dave and Fred swap their gifts. Then, Dave and Fred discuss something. Then, Action 28 repeats. Then, Dave and Alice swap their gifts. Then, Alice and Dave swap their gifts. Then, nothing happens for a while. Then, Fred and Claire swap their gifts (let's call it Action 44). Then, Action 34 repeats. Then, Action 35 repeats. Then, Bob and Claire swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Action 24 repeats. Then, Action 18 repeats. Then, Bob and Eve swap their gifts. Then, Eve and Fred swap their gifts. Then, nothing happens for a while. Then, Bob and Eve swap their gifts. Then, Eve and Bob swap their gifts. Then, nothing happens for a while. Then, Action 42 repeats. Then, Claire and Gertrude swap their gifts. Then, Action 14 repeats. Then, Dave and Fred swap their gifts. Then, Fred and Dave swap their gifts. Then, Dave and Eve swap their gifts. Then, Alice and Fred swap their gifts (let's call it Action 45). Then, Alice and Fred discuss something (let's call it Action 46). Then, Alice and Fred swap their gifts. Then, Action 1 repeats. Then, Gertrude and Dave swap their gifts. Then, Dave and Gertrude swap their gifts. Then, Gertrude and Eve swap their gifts. Then, Action 24 repeats. Then, Eve and Alice swap their gifts (let's call it Action 47). Then, Alice and Eve swap their gifts (let's call it Action 48). Then, Action 25 repeats. Then, nothing happens for a while. Then, Action 9 repeats. Then, Action 9 repeats. Then, Fred and Claire swap their gifts. Then, Claire and Fred swap their gifts (let's call it Action 49). Then, Alice and Fred swap their gifts. Then, Fred and Alice swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Claire swap their gifts. Then, Action 41 repeats. Then, Gertrude and Alice swap their gifts. Then, nothing happens for a while. Then, Eve and Bob swap their gifts. Then, Bob and Gertrude swap their gifts. Then, Gertrude and Bob swap their gifts. Then, Bob and Eve swap their gifts. Then, Action 25 repeats. Then, Eve and Fred discuss something (let's call it Action 50). Then, Eve and Fred swap their gifts. Then, Eve and Gertrude swap their gifts. Then, Action 22 repeats. Then, Bob and Gertrude swap their gifts. Then, Action 2 repeats. Then, Eve and Claire swap their gifts. Then, Eve and Claire discuss something. Then, Eve and Claire swap their gifts. Then, Action 43 repeats. Then, Action 34 repeats. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Gertrude and Dave swap their gifts. Then, Action 28 repeats. Then, Fred and Dave swap their gifts. Then, Dave and Gertrude swap their gifts. Then, Bob and Gertrude swap their gifts. Then, Gertrude and Bob swap their gifts. Then, Eve and Claire swap their gifts. Then, Eve and Claire discuss something. Then, Eve and Claire swap their gifts. Then, Eve and Dave swap their gifts. Then, Dave and Bob swap their gifts. Then, Bob and Dave swap their gifts. Then, Dave and Eve swap their gifts. Then, Action 16 repeats. Then, Bob and Alice swap their gifts. Then, Dave and Bob swap their gifts. Then, Bob and Fred swap their gifts (let's call it Action 51). Then, Fred and Bob swap their gifts (let's call it Action 52). Then, Action 48 repeats. Then, Alice and Eve discuss something (let's call it Action 53). Then, Alice and Eve swap their gifts. Then, nothing happens for a while. Then, Dave and Alice swap their gifts. Then, Alice and Bob swap their gifts. Then, Bob and Alice swap their gifts. Then, Alice and Dave swap their gifts. Then, Dave and Bob swap their gifts. Then, Dave and Bob discuss something (let's call it Action 54). Then, Dave and Bob swap their gifts. Then, Dave and Bob swap their gifts. Then, Action 19 repeats. Then, Action 16 repeats. Then, Action 32 repeats. Then, Gertrude and Bob swap their gifts. Then, Bob and Gertrude swap their gifts. Then, nothing happens for a while. Then, Gertrude and Eve swap their gifts. Then, Eve and Gertrude swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Bob swap their gifts. Then, Action 32 repeats. Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Gertrude swap their gifts. Then, Action 22 repeats. Then, Bob and Gertrude swap their gifts. Then, Action 4 repeats. Then, Action 9 repeats. Then, nothing happens for a while. Then, Bob and Dave swap their gifts. Then, Bob and Dave discuss something (let's call it Action 55). Then, Bob and Dave swap their gifts. Then, Dave and Alice swap their gifts. Then, Alice and Bob swap their gifts. Then, Bob and Alice swap their gifts. Then, Alice and Dave swap their gifts. Then, Fred and Claire swap their gifts. Then, Claire and Eve swap their gifts (let's call it Action 56). Then, Action 38 repeats. Then, Action 49 repeats. Then, Gertrude and Alice swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Action 20 repeats. Then, Dave and Fred swap their gifts. Then, Action 24 repeats. Then, Eve and Fred swap their gifts. Then, Fred and Dave swap their gifts. Then, Bob and Eve swap their gifts. Then, Eve and Fred swap their gifts. Then, Fred and Eve swap their gifts. Then, Eve and Bob swap their gifts. Then, Eve and Claire swap their gifts. Then, Action 56 repeats. Then, Action 4 repeats. Then, Gertrude and Alice discuss something. Then, Gertrude and Alice swap their gifts. Then, nothing happens for a while. Then, Action 51 repeats. Then, Action 52 repeats. Then, Action 32 repeats. Then, Bob and Dave discuss something. Then, Bob and Dave swap their gifts. Then, nothing happens for a while. Then, Bob and Gertrude swap their gifts. Then, Bob and Gertrude discuss something (let's call it Action 57). Then, Bob and Gertrude swap their gifts. Then, Fred and Dave swap their gifts. Then, Dave and Fred swap their gifts. Then, Dave and Alice swap their gifts. Then, Alice and Dave swap their gifts. Then, Action 38 repeats. Then, Eve and Claire discuss something. Then, Eve and Claire swap their gifts. Then, Gertrude and Bob swap their gifts. Then, Gertrude and Bob discuss something (let's call it Action 58). Then, Gertrude and Bob swap their gifts. Then, Action 22 repeats. Then, Bob and Gertrude swap their gifts. Then, Bob and Fred swap their gifts. Then, Fred and Bob swap their gifts. Then, Fred and Gertrude swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Fred swap their gifts. Then, nothing happens for a while. Then, Claire and Gertrude swap their gifts. Then, Gertrude and Claire swap their gifts. Then, Alice and Eve swap their gifts. Then, Eve and Alice swap their gifts. Then, Fred and Eve swap their gifts. Then, Eve and Fred swap their gifts. Then, Action 22 repeats. Then, Bob and Alice swap their gifts. Then, Alice and Bob swap their gifts. Then, Bob and Gertrude swap their gifts. Then, Eve and Bob swap their gifts. Then, Eve and Bob discuss something (let's call it Action 59). Then, Eve and Bob swap their gifts. Then, Dave and Alice swap their gifts. Then, Gertrude and Eve swap their gifts. Finally, Claire and Gertrude swap their gifts. At the end of the event, Claire has the\\nOptions:\\n(A) blue present\\n(B) green present\\n(C) brown present\\n(D) purple present\\n(E) orange ball\\n(F) pink ball\\n(G) red present\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}