{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"246937af-4731-5aed-9728-1c62285ec584","task_key":"bbeh~5fshuffled~5fobjects--full--246937af-4731-5aed-9728-1c62285ec584","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are on the same team in a…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are on the same team in a soccer match. At the start of the match, they are each assigned to a position: Alice is playing benchwarmer, Bob is playing cheerleader, Claire is playing left midfielder, Dave is playing center midfielder, Eve is playing left winger, Fred is playing right winger, and Gertrude is playing fullback.\\nAs the game progresses, pairs of players occasionally swap positions. First, Dave and Alice trade positions. Then, Bob and Eve trade positions (let's call it Action 1). Then, Bob and Eve discuss something (let's call it Action 2). Then, Bob and Eve trade positions. Then, Claire and Alice trade positions (let's call it Action 3). Then, Claire and Alice discuss something (let's call it Action 4). Then, Claire and Alice trade positions. Then, Fred and Eve trade positions (let's call it Action 5). Then, Fred and Eve discuss something (let's call it Action 6). Then, Action 5 repeats. Then, Fred and Alice trade positions (let's call it Action 7). Then, Alice and Fred trade positions (let's call it Action 8). Then, Eve and Dave trade positions (let's call it Action 9). Then, Dave and Claire trade positions (let's call it Action 10). Then, Claire and Dave trade positions (let's call it Action 11). Then, Dave and Eve trade positions (let's call it Action 12). Then, Bob and Claire trade positions (let's call it Action 13). Then, Claire and Dave trade positions. Then, Dave and Claire trade positions. Then, Claire and Bob trade positions (let's call it Action 14). Then, Eve and Alice trade positions (let's call it Action 15). Then, Alice and Eve trade positions (let's call it Action 16). Then, Dave and Claire trade positions. Then, Claire and Fred trade positions (let's call it Action 17). Then, Fred and Claire trade positions (let's call it Action 18). Then, Action 11 repeats. Then, nothing happens for a while (let's call it Action 19). Then, nothing happens for a while. Then, Bob and Fred trade positions (let's call it Action 20). Then, Fred and Alice trade positions. Then, Alice and Fred trade positions. Then, Fred and Bob trade positions (let's call it Action 21). Then, Dave and Fred trade positions (let's call it Action 22). Then, Fred and Dave trade positions (let's call it Action 23). Then, Bob and Gertrude trade positions (let's call it Action 24). Then, Gertrude and Eve trade positions (let's call it Action 25). Then, Eve and Gertrude trade positions (let's call it Action 26). Then, Gertrude and Bob trade positions (let's call it Action 27). Then, Gertrude and Bob trade positions. Then, Action 24 repeats. Then, Action 22 repeats. Then, Dave and Fred discuss something (let's call it Action 28). Then, Dave and Fred trade positions. Then, Gertrude and Alice trade positions (let's call it Action 29). Then, Gertrude and Alice discuss something (let's call it Action 30). Then, Gertrude and Alice trade positions. Then, Bob and Eve trade positions. Then, Eve and Bob trade positions (let's call it Action 31). Then, Gertrude and Alice trade positions. Then, Action 30 repeats. Then, Gertrude and Alice trade positions. Then, nothing happens for a while. Then, Action 19 repeats. Then, Action 22 repeats. Then, Action 28 repeats. Then, Dave and Fred trade positions. Then, Bob and Dave trade positions (let's call it Action 32). Then, Dave and Claire trade positions. Then, Action 11 repeats. Then, Dave and Bob trade positions (let's call it Action 33). Then, Alice and Gertrude trade positions (let's call it Action 34). Then, Gertrude and Alice trade positions. Then, Claire and Eve trade positions (let's call it Action 35). Then, Action 26 repeats. Then, Gertrude and Eve trade positions. Then, Eve and Claire trade positions (let's call it Action 36). Then, Action 24 repeats. Then, Gertrude and Bob trade positions. Then, Eve and Fred trade positions (let's call it Action 37). Then, Eve and Fred discuss something (let's call it Action 38). Then, Eve and Fred trade positions. Then, Action 19 repeats. Then, Claire and Bob trade positions. Then, Bob and Claire trade positions. Then, Dave and Alice trade positions (let's call it Action 39). Then, Dave and Alice discuss something (let's call it Action 40). Then, Action 39 repeats. Then, Dave and Bob trade positions. Then, Bob and Eve trade positions. Then, Action 31 repeats. Then, Bob and Dave trade positions. Then, Eve and Dave trade positions. Then, Dave and Eve trade positions. Then, Bob and Dave trade positions. Then, Dave and Bob trade positions. Then, nothing happens for a while. Then, Bob and Alice trade positions (let's call it Action 41). Then, Alice and Fred trade positions. Then, Fred and Alice trade positions. Then, Alice and Bob trade positions (let's call it Action 42). Then, Action 34 repeats. Then, Gertrude and Fred trade positions (let's call it Action 43). Then, Fred and Gertrude trade positions (let's call it Action 44). Then, Action 29 repeats. Then, Bob and Eve trade positions. Then, Eve and Bob trade positions. Then, nothing happens for a while. Then, Action 19 repeats. Then, nothing happens for a while. Then, Claire and Fred trade positions. Then, Fred and Dave trade positions. Then, Dave and Fred trade positions. Then, Fred and Claire trade positions. Then, Eve and Claire trade positions. Then, Eve and Claire discuss something (let's call it Action 45). Then, Eve and Claire trade positions. Then, Action 19 repeats. Then, Fred and Alice trade positions. Then, Fred and Alice discuss something (let's call it Action 46). Then, Fred and Alice trade positions. Then, nothing happens for a while. Then, Claire and Eve trade positions. Then, Eve and Dave trade positions. Then, Dave and Eve trade positions. Then, Eve and Claire trade positions. Then, Alice and Eve trade positions. Then, Eve and Alice trade positions. Then, Claire and Eve trade positions. Then, Eve and Gertrude trade positions. Then, Gertrude and Eve trade positions. Then, Eve and Claire trade positions. Then, Action 18 repeats. Then, nothing happens for a while. Then, Action 33 repeats. Then, Bob and Alice trade positions. Then, Alice and Bob trade positions. Then, Bob and Dave trade positions. Then, Action 20 repeats. Then, Fred and Gertrude trade positions. Then, Gertrude and Fred trade positions. Then, Fred and Bob trade positions. Then, Action 12 repeats. Then, Eve and Dave trade positions. Then, Dave and Gertrude trade positions (let's call it Action 47). Then, Action 27 repeats. Then, Bob and Gertrude trade positions. Then, Gertrude and Dave trade positions (let's call it Action 48). Then, Action 16 repeats. Then, Eve and Gertrude trade positions. Then, Gertrude and Eve trade positions. Then, Eve and Alice trade positions. Then, Action 14 repeats. Then, Claire and Bob discuss something (let's call it Action 49). Then, Action 14 repeats. Then, Action 19 repeats. Then, nothing happens for a while. Then, Claire and Fred trade positions. Then, Claire and Fred discuss something (let's call it Action 50). Then, Claire and Fred trade positions. Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Claire trade positions. Then, Claire and Gertrude trade positions (let's call it Action 51). Then, Gertrude and Claire trade positions (let's call it Action 52). Then, Claire and Dave trade positions. Then, Dave and Eve trade positions. Then, Eve and Dave trade positions. Then, Gertrude and Alice trade positions. Then, Gertrude and Alice discuss something. Then, Gertrude and Alice trade positions. Then, Bob and Eve trade positions. Then, Eve and Gertrude trade positions. Then, Gertrude and Eve trade positions. Then, Eve and Bob trade positions. Then, Fred and Bob trade positions. Then, Bob and Alice trade positions. Then, Alice and Bob trade positions. Then, Bob and Fred trade positions. Then, Dave and Claire trade positions. Then, Claire and Dave trade positions. Then, nothing happens for a while. Then, nothing happens for a while. Then, Gertrude and Dave trade positions. Then, Gertrude and Dave discuss something (let's call it Action 53). Then, Gertrude and Dave trade positions. Then, Bob and Gertrude trade positions. Then, Gertrude and Bob trade positions. Then, Claire and Dave trade positions. Then, Dave and Claire trade positions. Then, Fred and Gertrude trade positions. Then, Fred and Gertrude discuss something (let's call it Action 54). Then, Fred and Gertrude trade positions. Then, Gertrude and Eve trade positions. Then, Gertrude and Eve discuss something (let's call it Action 55). Then, Action 25 repeats. Then, Claire and Gertrude trade positions. Then, Gertrude and Claire trade positions. Then, Dave and Claire trade positions. Then, Dave and Claire discuss something (let's call it Action 56). Then, Dave and Claire trade positions. Then, Gertrude and Eve trade positions. Then, nothing happens for a while. Then, Fred and Gertrude trade positions. Then, Gertrude and Claire trade positions. Then, Claire and Gertrude trade positions. Then, Gertrude and Fred trade positions. Then, nothing happens for a while. Then, Dave and Fred trade positions. Then, Fred and Dave trade positions. Then, Bob and Alice trade positions. Then, Alice and Bob trade positions. Then, Bob and Dave trade positions. Then, Action 33 repeats. Then, Action 19 repeats. Then, Dave and Fred trade positions. Then, Fred and Dave trade positions. Then, Eve and Gertrude trade positions. Then, Eve and Gertrude discuss something (let's call it Action 57). Then, Action 26 repeats. Then, Action 22 repeats. Then, Dave and Fred discuss something. Then, Dave and Fred trade positions. Then, Claire and Bob trade positions. Then, Claire and Bob discuss something. Then, Claire and Bob trade positions. Then, Eve and Bob trade positions. Then, Eve and Bob discuss something (let's call it Action 58). Then, Eve and Bob trade positions. Then, Gertrude and Dave trade positions. Then, Gertrude and Dave discuss something. Then, Gertrude and Dave trade positions. Then, Gertrude and Eve trade positions. Then, Eve and Gertrude trade positions. Then, Gertrude and Eve trade positions. Then, Eve and Gertrude trade positions. Then, Eve and Gertrude trade positions. Then, Eve and Gertrude discuss something. Then, Eve and Gertrude trade positions. Then, nothing happens for a while. Then, Claire and Fred trade positions. Then, Fred and Claire trade positions. Then, nothing happens for a while. Then, Fred and Bob trade positions. Then, Bob and Gertrude trade positions. Then, Gertrude and Bob trade positions. Then, Action 20 repeats. Then, nothing happens for a while. Then, Action 10 repeats. Then, Claire and Gertrude trade positions. Then, Action 52 repeats. Then, Claire and Dave trade positions. Then, Claire and Dave trade positions. Then, Dave and Alice trade positions. Then, Alice and Dave trade positions (let's call it Action 59). Then, Dave and Claire trade positions. Then, Action 19 repeats. Then, Dave and Fred trade positions. Then, Fred and Dave trade positions. Then, Action 11 repeats. Then, Dave and Alice trade positions. Then, Alice and Dave trade positions. Then, Action 10 repeats. Then, Alice and Gertrude trade positions. Then, Gertrude and Alice trade positions. Then, Bob and Eve trade positions. Then, Eve and Bob trade positions. Then, nothing happens for a while. Then, Fred and Claire trade positions. Then, Claire and Fred trade positions. Then, Action 19 repeats. Then, Claire and Eve trade positions. Then, Eve and Fred trade positions. Then, Fred and Eve trade positions. Then, Action 36 repeats. Then, Bob and Alice trade positions. Then, Fred and Claire trade positions. Then, Action 51 repeats. Then, Gertrude and Claire trade positions. Then, Claire and Fred trade positions. Then, nothing happens for a while. Then, nothing happens for a while. Then, Gertrude and Claire trade positions. Then, Claire and Gertrude trade positions. Then, Bob and Eve trade positions. Then, Eve and Alice trade positions. Then, Action 16 repeats. Then, Eve and Bob trade positions. Then, Fred and Alice trade positions. Then, Alice and Fred trade positions. Then, Gertrude and Eve trade positions. Then, Eve and Gertrude trade positions. Then, Action 41 repeats. Then, Alice and Bob trade positions. Then, Bob and Fred trade positions. Then, Bob and Fred discuss something (let's call it Action 60). Then, Action 20 repeats. Then, Eve and Alice trade positions. Then, Alice and Eve trade positions. Then, Claire and Gertrude trade positions. Then, Gertrude and Claire trade positions. Then, Dave and Claire trade positions. Then, Claire and Fred trade positions. Then, Fred and Claire trade positions. Then, Claire and Dave trade positions. Then, Fred and Alice trade positions. Then, Action 8 repeats. Then, Action 3 repeats. Then, Claire and Alice discuss something. Then, Claire and Alice trade positions. Then, Eve and Claire trade positions. Then, Claire and Fred trade positions. Then, Fred and Claire trade positions. Then, Claire and Eve trade positions. Then, nothing happens for a while. Then, Action 17 repeats. Then, Action 7 repeats. Then, Alice and Fred trade positions. Then, Fred and Claire trade positions. Then, Action 14 repeats. Then, Bob and Claire trade positions. Then, Action 19 repeats. Then, nothing happens for a while. Then, Bob and Fred trade positions. Then, Bob and Fred discuss something. Then, Action 20 repeats. Then, Eve and Dave trade positions. Then, Dave and Eve trade positions. Then, Action 19 repeats. Then, Action 37 repeats. Then, Eve and Fred discuss something. Then, Action 37 repeats. Then, Alice and Eve trade positions. Then, Alice and Eve discuss something (let's call it Action 61). Then, Alice and Eve trade positions. Then, Action 19 repeats. Then, Action 19 repeats. Then, Action 51 repeats. Then, Gertrude and Claire trade positions. Then, Bob and Gertrude trade positions. Then, Gertrude and Alice trade positions. Finally, Eve and Bob trade positions. At the end of the match, Alice is playing\\nOptions:\\n(A) benchwarmer\\n(B) cheerleader\\n(C) left midfielder\\n(D) center midfielder\\n(E) left winger\\n(F) right winger\\n(G) fullback\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}