{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6a56af8a-2cc9-512e-ac88-36553ba8ab1b","task_key":"bbeh~5fshuffled~5fobjects--full--6a56af8a-2cc9-512e-ac88-36553ba8ab1b","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are on the same team in a…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are on the same team in a soccer match. At the start of the match, they are each assigned to a position: Alice is playing left winger, Bob is playing cheerleader, Claire is playing fullback, Dave is playing right winger, Eve is playing striker, Fred is playing right midfielder, and Gertrude is playing center midfielder.\\nAs the game progresses, pairs of players occasionally swap positions. First, Bob and Dave trade positions. Then, nothing happens for a while (let's call it Action 1). Then, Eve and Fred trade positions (let's call it Action 2). Then, Fred and Eve trade positions (let's call it Action 3). Then, Alice and Fred trade positions (let's call it Action 4). Then, Alice and Fred discuss something (let's call it Action 5). Then, Action 4 repeats. Then, Fred and Claire trade positions (let's call it Action 6). Then, Claire and Dave trade positions (let's call it Action 7). Then, Dave and Claire trade positions (let's call it Action 8). Then, Claire and Fred trade positions (let's call it Action 9). Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Claire trade positions. Then, Dave and Claire discuss something (let's call it Action 10). Then, Action 8 repeats. Then, Action 1 repeats. Then, Alice and Eve trade positions (let's call it Action 11). Then, Alice and Eve discuss something (let's call it Action 12). Then, Alice and Eve trade positions. Then, nothing happens for a while. Then, nothing happens for a while. Then, Gertrude and Eve trade positions (let's call it Action 13). Then, Eve and Gertrude trade positions (let's call it Action 14). Then, Claire and Fred trade positions. Then, Claire and Fred discuss something (let's call it Action 15). Then, Claire and Fred trade positions. Then, Gertrude and Alice trade positions (let's call it Action 16). Then, Alice and Gertrude trade positions (let's call it Action 17). Then, Claire and Bob trade positions (let's call it Action 18). Then, Claire and Bob discuss something (let's call it Action 19). Then, Claire and Bob trade positions. Then, Fred and Gertrude trade positions (let's call it Action 20). Then, Gertrude and Fred trade positions (let's call it Action 21). Then, Gertrude and Dave trade positions (let's call it Action 22). Then, Dave and Gertrude trade positions (let's call it Action 23). Then, Action 1 repeats. Then, nothing happens for a while. Then, nothing happens for a while. Then, Action 1 repeats. Then, Gertrude and Alice trade positions. Then, Alice and Gertrude trade positions. Then, nothing happens for a while. Then, Alice and Eve trade positions. Then, Eve and Alice trade positions (let's call it Action 24). Then, Bob and Alice trade positions (let's call it Action 25). Then, Alice and Bob trade positions (let's call it Action 26). Then, nothing happens for a while. Then, nothing happens for a while. Then, Eve and Claire trade positions (let's call it Action 27). Then, Eve and Fred trade positions. Then, Fred and Alice trade positions (let's call it Action 28). Then, Alice and Fred trade positions. Then, Fred and Eve trade positions. Then, Claire and Bob trade positions. Then, Bob and Claire trade positions (let's call it Action 29). Then, Bob and Dave trade positions (let's call it Action 30). Then, Dave and Bob trade positions (let's call it Action 31). Then, Action 24 repeats. Then, Eve and Alice discuss something (let's call it Action 32). Then, Eve and Alice trade positions. Then, nothing happens for a while. Then, nothing happens for a while. Then, Gertrude and Alice trade positions. Then, Alice and Claire trade positions (let's call it Action 33). Then, Claire and Alice trade positions (let's call it Action 34). Then, Action 17 repeats. Then, nothing happens for a while. Then, Fred and Bob trade positions (let's call it Action 35). Then, Bob and Claire trade positions. Then, Claire and Bob trade positions. Then, Bob and Fred trade positions (let's call it Action 36). Then, nothing happens for a while. Then, Gertrude and Fred trade positions. Then, Action 28 repeats. Then, Action 4 repeats. Then, Fred and Gertrude trade positions. Then, nothing happens for a while. Then, Action 30 repeats. Then, Dave and Gertrude trade positions. Then, Gertrude and Dave trade positions. Then, Dave and Bob trade positions. Then, Gertrude and Dave trade positions. Then, Dave and Bob trade positions. Then, Action 30 repeats. Then, Action 23 repeats. Then, Fred and Eve trade positions. Then, Fred and Eve discuss something (let's call it Action 37). Then, Fred and Eve trade positions. Then, Claire and Fred trade positions. Then, Fred and Claire trade positions. Then, Bob and Gertrude trade positions (let's call it Action 38). Then, Action 22 repeats. Then, Dave and Gertrude trade positions. Then, Gertrude and Bob trade positions (let's call it Action 39). Then, Alice and Gertrude trade positions. Then, Gertrude and Fred trade positions. Then, Fred and Gertrude trade positions. Then, Action 16 repeats. Then, Action 36 repeats. Then, Action 35 repeats. Then, Action 1 repeats. Then, Dave and Eve trade positions (let's call it Action 40). Then, Eve and Dave trade positions (let's call it Action 41). Then, Gertrude and Alice trade positions. Then, Action 17 repeats. Then, Action 21 repeats. Then, Fred and Gertrude trade positions. Then, Gertrude and Alice trade positions. Then, Fred and Gertrude trade positions. Then, Gertrude and Fred trade positions. Then, Fred and Alice trade positions. Then, Action 4 repeats. Then, nothing happens for a while. Then, Bob and Alice trade positions. Then, Bob and Alice discuss something (let's call it Action 42). Then, Bob and Alice trade positions. Then, Claire and Alice trade positions. Then, Action 33 repeats. Then, Claire and Eve trade positions (let's call it Action 43). Then, Eve and Dave trade positions. Then, Dave and Eve trade positions. Then, Eve and Claire trade positions. Then, nothing happens for a while. Then, Eve and Dave trade positions. Then, Eve and Dave discuss something (let's call it Action 44). Then, Eve and Dave trade positions. Then, Eve and Alice trade positions. Then, Action 4 repeats. Then, Action 28 repeats. Then, Alice and Eve trade positions. Then, Gertrude and Dave trade positions. Then, Gertrude and Dave discuss something (let's call it Action 45). Then, Action 22 repeats. Then, Action 18 repeats. Then, Bob and Alice trade positions. Then, Action 26 repeats. Then, Action 29 repeats. Then, Action 4 repeats. Then, Fred and Alice trade positions. Then, Gertrude and Eve trade positions. Then, Eve and Gertrude trade positions. Then, Bob and Alice trade positions. Then, Alice and Bob trade positions. Then, nothing happens for a while. Then, nothing happens for a while. Then, Gertrude and Bob trade positions. Then, Bob and Dave trade positions. Then, Dave and Bob trade positions. Then, Bob and Gertrude trade positions. Then, Action 1 repeats. Then, Action 7 repeats. Then, Dave and Claire trade positions. Then, Bob and Alice trade positions. Then, Action 11 repeats. Then, Eve and Alice trade positions. Then, Alice and Bob trade positions. Then, Dave and Alice trade positions (let's call it Action 46). Then, Alice and Bob trade positions. Then, Bob and Alice trade positions. Then, Alice and Dave trade positions (let's call it Action 47). Then, Eve and Claire trade positions. Then, Eve and Claire discuss something (let's call it Action 48). Then, Eve and Claire trade positions. Then, Dave and Gertrude trade positions. Then, Gertrude and Claire trade positions (let's call it Action 49). Then, Claire and Gertrude trade positions (let's call it Action 50). Then, Gertrude and Dave trade positions. Then, Gertrude and Claire trade positions. Then, Claire and Gertrude trade positions. Then, Fred and Gertrude trade positions. Then, Gertrude and Eve trade positions. Then, Action 14 repeats. Then, Gertrude and Fred trade positions. Then, Bob and Gertrude trade positions. Then, Gertrude and Bob trade positions. Then, Alice and Fred trade positions. Then, Action 20 repeats. Then, Gertrude and Fred trade positions. Then, Fred and Alice trade positions. Then, Eve and Claire trade positions. Then, Action 48 repeats. Then, Eve and Claire trade positions. Then, Eve and Bob trade positions (let's call it Action 51). Then, Eve and Bob discuss something (let's call it Action 52). Then, Eve and Bob trade positions. Then, nothing happens for a while. Then, Gertrude and Eve trade positions. Then, Eve and Gertrude trade positions. Then, nothing happens for a while. Then, Claire and Gertrude trade positions. Then, Gertrude and Claire trade positions. Then, Action 9 repeats. Then, Action 15 repeats. Then, Action 9 repeats. Then, nothing happens for a while. Then, Action 1 repeats. Then, Dave and Fred trade positions (let's call it Action 53). Then, Fred and Gertrude trade positions. Then, Gertrude and Fred trade positions. Then, Fred and Dave trade positions (let's call it Action 54). Then, Gertrude and Bob trade positions. Then, Bob and Gertrude trade positions. Then, nothing happens for a while. Then, Action 38 repeats. Then, Bob and Gertrude discuss something (let's call it Action 55). Then, Action 38 repeats. Then, Bob and Gertrude trade positions. Then, Gertrude and Fred trade positions. Then, Fred and Gertrude trade positions. Then, Gertrude and Bob trade positions. Then, Fred and Bob trade positions. Then, Fred and Bob discuss something (let's call it Action 56). Then, Fred and Bob trade positions. Then, Gertrude and Dave trade positions. Then, Action 23 repeats. Then, Bob and Claire trade positions. Then, nothing happens for a while. Then, Action 17 repeats. Then, Gertrude and Alice trade positions. Then, Eve and Bob trade positions. Then, Bob and Eve trade positions (let's call it Action 57). Then, nothing happens for a while. Then, Fred and Eve trade positions. Then, Fred and Eve discuss something. Then, Action 3 repeats. Then, Action 13 repeats. Then, Eve and Claire trade positions. Then, Claire and Eve trade positions. Then, Eve and Gertrude trade positions. Then, nothing happens for a while. Then, Fred and Bob trade positions. Then, Bob and Alice trade positions. Then, Action 26 repeats. Then, Bob and Fred trade positions. Then, Bob and Alice trade positions. Then, Alice and Fred trade positions. Then, Action 28 repeats. Then, Alice and Bob trade positions. Then, nothing happens for a while. Then, Dave and Bob trade positions. Then, Action 36 repeats. Then, Fred and Bob trade positions. Then, Action 30 repeats. Then, Claire and Bob trade positions. Then, Claire and Bob discuss something. Then, Claire and Bob trade positions. Then, Claire and Eve trade positions. Then, Action 27 repeats. Then, Bob and Eve trade positions. Then, Action 51 repeats. Then, Eve and Fred trade positions. Then, Eve and Fred discuss something (let's call it Action 58). Then, Action 2 repeats. Then, Alice and Eve trade positions. Then, Action 12 repeats. Then, Alice and Eve trade positions. Then, Bob and Alice trade positions. Then, Bob and Alice discuss something. Then, Bob and Alice trade positions. Then, Fred and Eve trade positions. Then, Eve and Bob trade positions. Then, Bob and Eve trade positions. Then, Action 2 repeats. Then, Claire and Dave trade positions. Then, Action 46 repeats. Then, Action 47 repeats. Then, Dave and Claire trade positions. Then, Eve and Dave trade positions. Then, Eve and Dave discuss something. Then, Eve and Dave trade positions. Then, Action 1 repeats. Then, Alice and Dave trade positions. Then, Dave and Gertrude trade positions. Then, Action 22 repeats. Then, Action 46 repeats. Then, Action 14 repeats. Then, Gertrude and Claire trade positions. Then, Action 50 repeats. Then, Gertrude and Eve trade positions. Then, Alice and Bob trade positions. Then, Action 25 repeats. Then, Fred and Claire trade positions. Then, Claire and Fred trade positions. Then, Dave and Gertrude trade positions. Then, Dave and Gertrude discuss something (let's call it Action 59). Then, Action 23 repeats. Then, Action 14 repeats. Then, Gertrude and Eve trade positions. Then, Alice and Bob trade positions. Then, Eve and Fred trade positions. Finally, Alice and Fred trade positions. At the end of the match, Fred is playing\\nOptions:\\n(A) left winger\\n(B) cheerleader\\n(C) fullback\\n(D) right winger\\n(E) striker\\n(F) right midfielder\\n(G) center midfielder\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}