{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"436af2bb-c456-5c38-b513-c9abdec2b42c","task_key":"bbeh~5fshuffled~5fobjects--full--436af2bb-c456-5c38-b513-c9abdec2b42c","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are on the same team in a…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are on the same team in a soccer match. At the start of the match, they are each assigned to a position: Alice is playing striker, Bob is playing goalkeeper, Claire is playing left winger, Dave is playing benchwarmer, Eve is playing center midfielder, Fred is playing fullback, and Gertrude is playing cheerleader.\\nAs the game progresses, pairs of players occasionally swap positions. First, Alice and Fred trade positions. Then, Claire and Dave trade positions (let's call it Action 1). Then, Claire and Dave discuss something (let's call it Action 2). Then, Action 1 repeats. Then, Bob and Eve trade positions (let's call it Action 3). Then, Eve and Gertrude trade positions (let's call it Action 4). Then, Gertrude and Eve trade positions (let's call it Action 5). Then, Eve and Bob trade positions (let's call it Action 6). Then, Bob and Eve trade positions. Then, Eve and Bob trade positions. Then, Alice and Bob trade positions (let's call it Action 7). Then, Bob and Gertrude trade positions (let's call it Action 8). Then, Gertrude and Bob trade positions (let's call it Action 9). Then, Bob and Alice trade positions (let's call it Action 10). Then, nothing happens for a while (let's call it Action 11). Then, Bob and Alice trade positions. Then, Alice and Claire trade positions (let's call it Action 12). Then, Claire and Alice trade positions (let's call it Action 13). Then, Alice and Bob trade positions. Then, Action 11 repeats. Then, nothing happens for a while. Then, Eve and Alice trade positions (let's call it Action 14). Then, Alice and Claire trade positions. Then, Claire and Alice trade positions. Then, Alice and Eve trade positions (let's call it Action 15). Then, Claire and Fred trade positions (let's call it Action 16). Then, Claire and Fred discuss something (let's call it Action 17). Then, Claire and Fred trade positions. Then, Fred and Gertrude trade positions (let's call it Action 18). Then, Fred and Gertrude discuss something (let's call it Action 19). Then, Fred and Gertrude trade positions. Then, Action 15 repeats. Then, Action 14 repeats. Then, Fred and Claire trade positions (let's call it Action 20). Then, Claire and Fred trade positions. Then, Eve and Dave trade positions (let's call it Action 21). Then, Eve and Dave discuss something (let's call it Action 22). Then, Eve and Dave trade positions. Then, Action 11 repeats. Then, Alice and Eve trade positions. Then, Eve and Alice trade positions. Then, nothing happens for a while. Then, Alice and Fred trade positions (let's call it Action 23). Then, Fred and Dave trade positions (let's call it Action 24). Then, Dave and Fred trade positions (let's call it Action 25). Then, Fred and Alice trade positions (let's call it Action 26). Then, Claire and Fred trade positions. Then, Fred and Eve trade positions (let's call it Action 27). Then, Eve and Fred trade positions (let's call it Action 28). Then, Fred and Claire trade positions. Then, nothing happens for a while. Then, Claire and Dave trade positions. Then, Dave and Gertrude trade positions (let's call it Action 29). Then, Gertrude and Dave trade positions (let's call it Action 30). Then, Dave and Claire trade positions (let's call it Action 31). Then, Fred and Dave trade positions. Then, Dave and Fred trade positions. Then, nothing happens for a while. Then, Action 23 repeats. Then, Fred and Gertrude trade positions. Then, Gertrude and Fred trade positions (let's call it Action 32). Then, Fred and Alice trade positions. Then, Gertrude and Fred trade positions. Then, Fred and Eve trade positions. Then, Eve and Fred trade positions. Then, Fred and Gertrude trade positions. Then, nothing happens for a while. Then, Gertrude and Alice trade positions (let's call it Action 33). Then, Gertrude and Alice discuss something (let's call it Action 34). Then, Action 33 repeats. Then, Alice and Claire trade positions. Then, Claire and Gertrude trade positions (let's call it Action 35). Then, Gertrude and Claire trade positions (let's call it Action 36). Then, Claire and Alice trade positions. Then, Action 11 repeats. Then, Action 28 repeats. Then, Eve and Fred discuss something (let's call it Action 37). Then, Eve and Fred trade positions. Then, Action 11 repeats. Then, Action 10 repeats. Then, Bob and Alice discuss something (let's call it Action 38). Then, Action 10 repeats. Then, Bob and Claire trade positions (let's call it Action 39). Then, Claire and Eve trade positions (let's call it Action 40). Then, Eve and Claire trade positions (let's call it Action 41). Then, Claire and Bob trade positions (let's call it Action 42). Then, Claire and Alice trade positions. Then, Action 12 repeats. Then, Alice and Bob trade positions. Then, Bob and Alice trade positions. Then, Bob and Dave trade positions (let's call it Action 43). Then, Bob and Dave discuss something (let's call it Action 44). Then, Bob and Dave trade positions. Then, nothing happens for a while. Then, Fred and Alice trade positions. Then, Action 12 repeats. Then, Action 13 repeats. Then, Alice and Fred trade positions. Then, Action 33 repeats. Then, Action 23 repeats. Then, Action 26 repeats. Then, Alice and Gertrude trade positions (let's call it Action 45). Then, Claire and Fred trade positions. Then, Fred and Claire trade positions. Then, nothing happens for a while. Then, Gertrude and Eve trade positions. Then, Eve and Dave trade positions. Then, Dave and Eve trade positions (let's call it Action 46). Then, Action 4 repeats. Then, nothing happens for a while. Then, Action 11 repeats. Then, Gertrude and Fred trade positions. Then, Fred and Bob trade positions (let's call it Action 47). Then, Bob and Fred trade positions (let's call it Action 48). Then, Fred and Gertrude trade positions. Then, Gertrude and Bob trade positions. Then, Gertrude and Bob discuss something (let's call it Action 49). Then, Gertrude and Bob trade positions. Then, Alice and Fred trade positions. Then, Fred and Bob trade positions. Then, Bob and Fred trade positions. Then, Fred and Alice trade positions. Then, Dave and Claire trade positions. Then, Dave and Claire discuss something (let's call it Action 50). Then, Dave and Claire trade positions. Then, Alice and Eve trade positions. Then, Alice and Eve discuss something (let's call it Action 51). Then, Alice and Eve trade positions. Then, Bob and Eve trade positions. Then, Eve and Fred trade positions. Then, Fred and Eve trade positions. Then, Eve and Bob trade positions. Then, Action 35 repeats. Then, Action 11 repeats. Then, Bob and Claire trade positions. Then, Action 42 repeats. Then, Bob and Dave trade positions. Then, Bob and Dave discuss something. Then, Action 43 repeats. Then, Bob and Alice trade positions. Then, Alice and Bob trade positions. Then, Eve and Fred trade positions. Then, Action 27 repeats. Then, Bob and Dave trade positions. Then, Dave and Eve trade positions. Then, Eve and Dave trade positions. Then, Dave and Bob trade positions (let's call it Action 52). Then, Action 12 repeats. Then, Alice and Claire discuss something (let's call it Action 53). Then, Alice and Claire trade positions. Then, Action 46 repeats. Then, Dave and Eve discuss something (let's call it Action 54). Then, Dave and Eve trade positions. Then, Gertrude and Fred trade positions. Then, Gertrude and Fred discuss something (let's call it Action 55). Then, Action 32 repeats. Then, Action 36 repeats. Then, Gertrude and Claire discuss something (let's call it Action 56). Then, Gertrude and Claire trade positions. Then, Action 27 repeats. Then, Fred and Eve discuss something (let's call it Action 57). Then, Fred and Eve trade positions. Then, Eve and Claire trade positions. Then, Claire and Eve trade positions. Then, Action 40 repeats. Then, Eve and Gertrude trade positions. Then, Action 5 repeats. Then, Eve and Claire trade positions. Then, nothing happens for a while. Then, Eve and Claire trade positions. Then, Action 40 repeats. Then, Alice and Eve trade positions. Then, Eve and Alice trade positions. Then, Gertrude and Claire trade positions. Then, Action 13 repeats. Then, Alice and Claire trade positions. Then, Claire and Gertrude trade positions. Then, Fred and Dave trade positions. Then, Dave and Fred trade positions. Then, Alice and Eve trade positions. Then, Eve and Fred trade positions. Then, Fred and Eve trade positions. Then, Action 14 repeats. Then, nothing happens for a while. Then, Dave and Alice trade positions (let's call it Action 58). Then, Action 16 repeats. Then, Fred and Dave trade positions. Then, Dave and Fred trade positions. Then, Fred and Claire trade positions. Then, Alice and Eve trade positions. Then, Eve and Bob trade positions. Then, Bob and Eve trade positions. Then, Eve and Alice trade positions. Then, Gertrude and Claire trade positions. Then, Claire and Gertrude trade positions. Then, Dave and Fred trade positions. Then, Fred and Claire trade positions. Then, Action 16 repeats. Then, Fred and Dave trade positions. Then, Gertrude and Dave trade positions. Then, Dave and Gertrude trade positions. Then, Gertrude and Claire trade positions. Then, Gertrude and Claire discuss something. Then, Action 36 repeats. Then, Action 11 repeats. Then, Action 11 repeats. Then, nothing happens for a while. Then, Action 58 repeats. Then, Alice and Fred trade positions. Then, Fred and Alice trade positions. Then, Alice and Dave trade positions (let's call it Action 59). Then, nothing happens for a while. Then, Bob and Gertrude trade positions. Then, Gertrude and Bob trade positions. Then, Claire and Alice trade positions. Then, Alice and Gertrude trade positions. Then, Gertrude and Alice trade positions. Then, Action 12 repeats. Then, nothing happens for a while. Then, Bob and Claire trade positions. Then, Action 42 repeats. Then, Fred and Alice trade positions. Then, Action 23 repeats. Then, Gertrude and Dave trade positions. Then, Dave and Alice trade positions. Then, Alice and Dave trade positions. Then, Dave and Gertrude trade positions. Then, nothing happens for a while. Then, Bob and Dave trade positions. Then, Bob and Dave discuss something. Then, Bob and Dave trade positions. Then, Action 43 repeats. Then, Dave and Eve trade positions. Then, Action 21 repeats. Then, Action 52 repeats. Then, nothing happens for a while. Then, Action 10 repeats. Then, Alice and Bob trade positions. Then, Dave and Fred trade positions. Then, Dave and Fred discuss something (let's call it Action 60). Then, Dave and Fred trade positions. Then, Action 15 repeats. Then, Eve and Bob trade positions. Then, Bob and Eve trade positions. Then, Eve and Alice trade positions. Then, Action 6 repeats. Then, Eve and Bob discuss something (let's call it Action 61). Then, Eve and Bob trade positions. Then, Action 11 repeats. Then, Claire and Gertrude trade positions. Then, Claire and Gertrude discuss something (let's call it Action 62). Then, Claire and Gertrude trade positions. Then, Eve and Fred trade positions. Then, Fred and Bob trade positions. Then, Bob and Fred trade positions. Then, Fred and Eve trade positions. Then, Action 11 repeats. Then, Alice and Gertrude trade positions. Then, Alice and Gertrude discuss something (let's call it Action 63). Then, Alice and Gertrude trade positions. Then, Action 29 repeats. Then, Gertrude and Claire trade positions. Then, Claire and Gertrude trade positions. Then, Gertrude and Dave trade positions. Then, Bob and Dave trade positions. Then, Action 10 repeats. Then, Alice and Bob trade positions. Then, nothing happens for a while. Then, Claire and Eve trade positions. Then, Eve and Bob trade positions. Then, Bob and Eve trade positions. Then, Eve and Claire trade positions. Then, Gertrude and Claire trade positions. Then, Claire and Gertrude trade positions. Then, Action 11 repeats. Then, Eve and Gertrude trade positions. Then, Eve and Gertrude discuss something (let's call it Action 64). Then, Action 4 repeats. Then, Action 21 repeats. Then, Action 25 repeats. Then, Fred and Dave trade positions. Then, Dave and Eve trade positions. Then, Eve and Gertrude trade positions. Then, Eve and Gertrude discuss something. Then, Eve and Gertrude trade positions. Then, Action 30 repeats. Then, Dave and Alice trade positions. Then, Alice and Dave trade positions. Then, Dave and Gertrude trade positions. Then, Eve and Alice trade positions. Then, Alice and Eve trade positions. Then, Claire and Fred trade positions. Then, Action 17 repeats. Then, Claire and Fred trade positions. Then, Action 43 repeats. Then, Dave and Eve trade positions. Then, Eve and Dave trade positions. Then, Dave and Bob trade positions. Then, Dave and Eve trade positions. Then, Dave and Eve discuss something. Then, Dave and Eve trade positions. Then, nothing happens for a while. Then, Eve and Dave trade positions. Then, Eve and Dave discuss something. Then, Eve and Dave trade positions. Then, Claire and Gertrude trade positions. Then, Claire and Gertrude discuss something. Then, Claire and Gertrude trade positions. Then, Gertrude and Alice trade positions. Then, Alice and Eve trade positions. Then, Eve and Alice trade positions. Then, Action 45 repeats. Then, Action 18 repeats. Then, Fred and Gertrude discuss something. Then, Fred and Gertrude trade positions. Then, Claire and Bob trade positions. Then, Bob and Dave trade positions. Then, Action 52 repeats. Then, Bob and Claire trade positions. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Eve and Dave trade positions. Then, Action 46 repeats. Then, Action 11 repeats. Then, Fred and Eve trade positions. Then, Eve and Bob trade positions. Then, Bob and Eve trade positions. Then, Eve and Fred trade positions. Then, nothing happens for a while. Then, Action 3 repeats. Then, Eve and Alice trade positions. Finally, Gertrude and Dave trade positions. At the end of the match, Bob is playing\\nOptions:\\n(A) striker\\n(B) goalkeeper\\n(C) left winger\\n(D) benchwarmer\\n(E) center midfielder\\n(F) fullback\\n(G) cheerleader\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}