{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fac9df52-1687-59c1-b1fd-a43bac0971ae","task_key":"bbeh~5fshuffled~5fobjects--full--fac9df52-1687-59c1-b1fd-a43bac0971ae","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are on the same team in a…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are on the same team in a soccer match. At the start of the match, they are each assigned to a position: Alice is playing striker, Bob is playing right winger, Claire is playing left winger, Dave is playing benchwarmer, Eve is playing goalkeeper, Fred is playing center midfielder, and Gertrude is playing cheerleader.\\nAs the game progresses, pairs of players occasionally swap positions. First, Eve and Claire trade positions. Then, Gertrude and Eve trade positions (let's call it Action 1). Then, Eve and Fred trade positions (let's call it Action 2). Then, Fred and Eve trade positions (let's call it Action 3). Then, Eve and Gertrude trade positions (let's call it Action 4). Then, Eve and Alice trade positions (let's call it Action 5). Then, Eve and Alice discuss something (let's call it Action 6). Then, Eve and Alice trade positions. Then, Gertrude and Bob trade positions (let's call it Action 7). Then, Bob and Alice trade positions (let's call it Action 8). Then, Alice and Bob trade positions (let's call it Action 9). Then, Bob and Gertrude trade positions (let's call it Action 10). Then, Eve and Claire trade positions (let's call it Action 11). Then, Eve and Claire discuss something (let's call it Action 12). Then, Action 11 repeats. Then, Claire and Bob trade positions (let's call it Action 13). Then, Bob and Claire trade positions (let's call it Action 14). Then, nothing happens for a while (let's call it Action 15). Then, Eve and Gertrude trade positions. Then, Eve and Gertrude discuss something (let's call it Action 16). Then, Eve and Gertrude trade positions. Then, Claire and Fred trade positions (let's call it Action 17). Then, Fred and Claire trade positions (let's call it Action 18). Then, nothing happens for a while. Then, Dave and Alice trade positions (let's call it Action 19). Then, Dave and Alice discuss something (let's call it Action 20). Then, Dave and Alice trade positions. Then, nothing happens for a while. Then, Dave and Bob trade positions (let's call it Action 21). Then, Dave and Bob discuss something (let's call it Action 22). Then, Action 21 repeats. Then, Action 5 repeats. Then, Alice and Eve trade positions (let's call it Action 23). Then, Gertrude and Alice trade positions (let's call it Action 24). Then, Alice and Bob trade positions. Then, Action 8 repeats. Then, Alice and Gertrude trade positions (let's call it Action 25). Then, nothing happens for a while. Then, nothing happens for a while. Then, Action 15 repeats. Then, Fred and Gertrude trade positions (let's call it Action 26). Then, Gertrude and Claire trade positions (let's call it Action 27). Then, Claire and Gertrude trade positions (let's call it Action 28). Then, Gertrude and Fred trade positions (let's call it Action 29). Then, Action 7 repeats. Then, Bob and Claire trade positions. Then, Claire and Bob trade positions. Then, Bob and Gertrude trade positions. Then, Action 18 repeats. Then, Fred and Claire discuss something (let's call it Action 30). Then, Fred and Claire trade positions. Then, Action 11 repeats. Then, Claire and Eve trade positions (let's call it Action 31). Then, Action 27 repeats. Then, Claire and Gertrude trade positions. Then, Alice and Fred trade positions (let's call it Action 32). Then, Fred and Bob trade positions (let's call it Action 33). Then, Bob and Fred trade positions (let's call it Action 34). Then, Fred and Alice trade positions (let's call it Action 35). Then, Action 9 repeats. Then, Bob and Gertrude trade positions. Then, Gertrude and Bob trade positions. Then, Bob and Alice trade positions. Then, Fred and Claire trade positions. Then, Fred and Claire discuss something. Then, Action 18 repeats. Then, Action 24 repeats. Then, Gertrude and Alice discuss something (let's call it Action 36). Then, Action 24 repeats. Then, Gertrude and Alice trade positions. Then, Alice and Gertrude trade positions. Then, Dave and Fred trade positions (let's call it Action 37). Then, Dave and Fred discuss something (let's call it Action 38). Then, Action 37 repeats. Then, Claire and Dave trade positions (let's call it Action 39). Then, Action 21 repeats. Then, Bob and Dave trade positions (let's call it Action 40). Then, Dave and Claire trade positions (let's call it Action 41). Then, Action 14 repeats. Then, Bob and Claire discuss something (let's call it Action 42). Then, Bob and Claire trade positions. Then, Alice and Dave trade positions (let's call it Action 43). Then, Dave and Alice trade positions. Then, Action 37 repeats. Then, Dave and Fred discuss something. Then, Dave and Fred trade positions. Then, Action 15 repeats. Then, Bob and Fred trade positions. Then, Fred and Bob trade positions. Then, nothing happens for a while. Then, Gertrude and Dave trade positions (let's call it Action 44). Then, Dave and Alice trade positions. Then, Alice and Dave trade positions. Then, Dave and Gertrude trade positions (let's call it Action 45). Then, Eve and Claire trade positions. Then, Action 12 repeats. Then, Eve and Claire trade positions. Then, Fred and Dave trade positions (let's call it Action 46). Then, Dave and Fred trade positions. Then, Eve and Bob trade positions (let's call it Action 47). Then, Bob and Dave trade positions. Then, Action 21 repeats. Then, Bob and Eve trade positions (let's call it Action 48). Then, Fred and Gertrude trade positions. Then, Gertrude and Eve trade positions. Then, Eve and Gertrude trade positions. Then, Gertrude and Fred trade positions. Then, nothing happens for a while. Then, Eve and Fred trade positions. Then, Eve and Fred discuss something (let's call it Action 49). Then, Eve and Fred trade positions. Then, Eve and Gertrude trade positions. Then, Gertrude and Bob trade positions. Then, Bob and Gertrude trade positions. Then, Gertrude and Eve trade positions. Then, Bob and Fred trade positions. Then, Fred and Bob trade positions. Then, Alice and Gertrude trade positions. Then, Alice and Gertrude discuss something (let's call it Action 50). Then, Action 25 repeats. Then, Claire and Bob trade positions. Then, Bob and Claire trade positions. Then, Dave and Fred trade positions. Then, Dave and Fred discuss something. Then, Dave and Fred trade positions. Then, Gertrude and Alice trade positions. Then, Gertrude and Dave trade positions. Then, Action 45 repeats. Then, Alice and Dave trade positions. Then, Dave and Alice trade positions. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Eve and Dave trade positions (let's call it Action 51). Then, Dave and Eve trade positions (let's call it Action 52). Then, nothing happens for a while. Then, Alice and Bob trade positions. Then, Action 34 repeats. Then, Action 33 repeats. Then, Bob and Alice trade positions. Then, nothing happens for a while. Then, nothing happens for a while. Then, Bob and Gertrude trade positions. Then, Gertrude and Bob trade positions. Then, nothing happens for a while. Then, Bob and Gertrude trade positions. Then, Gertrude and Fred trade positions. Then, Fred and Gertrude trade positions. Then, Gertrude and Bob trade positions. Then, Action 15 repeats. Then, Action 2 repeats. Then, Fred and Bob trade positions. Then, Bob and Fred trade positions. Then, Fred and Eve trade positions. Then, Action 15 repeats. Then, Alice and Gertrude trade positions. Then, Gertrude and Alice trade positions. Then, Claire and Alice trade positions (let's call it Action 53). Then, Claire and Alice discuss something (let's call it Action 54). Then, Claire and Alice trade positions. Then, Alice and Dave trade positions. Then, Dave and Alice trade positions. Then, Alice and Dave trade positions. Then, Dave and Fred trade positions. Then, Fred and Dave trade positions. Then, Dave and Alice trade positions. Then, Action 48 repeats. Then, Eve and Alice trade positions. Then, Action 23 repeats. Then, Eve and Bob trade positions. Then, Gertrude and Alice trade positions. Then, Alice and Claire trade positions (let's call it Action 55). Then, Claire and Alice trade positions. Then, Alice and Gertrude trade positions. Then, Gertrude and Fred trade positions. Then, Fred and Gertrude trade positions. Then, Fred and Bob trade positions. Then, Gertrude and Claire trade positions. Then, Claire and Gertrude trade positions. Then, Eve and Bob trade positions. Then, Eve and Bob discuss something (let's call it Action 56). Then, Eve and Bob trade positions. Then, Action 15 repeats. Then, Fred and Bob trade positions. Then, Action 10 repeats. Then, Gertrude and Bob trade positions. Then, Bob and Fred trade positions. Then, Fred and Alice trade positions. Then, Alice and Dave trade positions. Then, Action 19 repeats. Then, Alice and Fred trade positions. Then, Fred and Eve trade positions. Then, Fred and Eve discuss something (let's call it Action 57). Then, Action 3 repeats. Then, Gertrude and Dave trade positions. Then, Dave and Bob trade positions. Then, Bob and Dave trade positions. Then, Action 45 repeats. Then, Eve and Gertrude trade positions. Then, Gertrude and Eve trade positions. Then, Action 15 repeats. Then, Fred and Claire trade positions. Then, Action 31 repeats. Then, Eve and Claire trade positions. Then, Action 17 repeats. Then, nothing happens for a while. Then, Bob and Claire trade positions. Then, Bob and Claire discuss something. Then, Action 14 repeats. Then, Claire and Dave trade positions. Then, Dave and Eve trade positions. Then, Action 51 repeats. Then, Action 41 repeats. Then, Eve and Gertrude trade positions. Then, Eve and Gertrude discuss something. Then, Eve and Gertrude trade positions. Then, Fred and Eve trade positions. Then, Eve and Fred trade positions. Then, Fred and Alice trade positions. Then, Alice and Gertrude trade positions. Then, Gertrude and Alice trade positions. Then, Alice and Fred trade positions. Then, nothing happens for a while. Then, Fred and Eve trade positions. Then, Eve and Fred trade positions. Then, Bob and Claire trade positions. Then, Claire and Bob trade positions. Then, Action 45 repeats. Then, Gertrude and Alice trade positions. Then, Alice and Gertrude trade positions. Then, Action 44 repeats. Then, Alice and Eve trade positions. Then, Eve and Gertrude trade positions. Then, Action 1 repeats. Then, Eve and Alice trade positions. Then, Gertrude and Dave trade positions. Then, Dave and Fred trade positions. Then, Fred and Dave trade positions. Then, Action 45 repeats. Then, Dave and Fred trade positions. Then, Action 46 repeats. Then, Fred and Bob trade positions. Then, Fred and Bob discuss something (let's call it Action 58). Then, Fred and Bob trade positions. Then, Eve and Alice trade positions. Then, Action 25 repeats. Then, Gertrude and Alice trade positions. Then, Alice and Eve trade positions. Then, nothing happens for a while. Then, nothing happens for a while. Then, Claire and Eve trade positions. Then, Eve and Alice trade positions. Then, Alice and Eve trade positions. Then, Action 11 repeats. Then, nothing happens for a while. Then, Bob and Dave trade positions. Then, Action 37 repeats. Then, Action 46 repeats. Then, Action 21 repeats. Then, Alice and Eve trade positions. Then, Action 5 repeats. Then, Claire and Fred trade positions. Then, Fred and Bob trade positions. Then, Bob and Fred trade positions. Then, Fred and Claire trade positions. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Dave trade positions. Then, Dave and Alice trade positions. Then, Action 7 repeats. Then, Bob and Gertrude trade positions. Then, Fred and Bob trade positions. Then, Action 58 repeats. Then, Fred and Bob trade positions. Then, Action 15 repeats. Then, Gertrude and Eve trade positions. Then, Eve and Fred trade positions. Then, Action 3 repeats. Then, Action 4 repeats. Then, nothing happens for a while. Then, Dave and Fred trade positions. Then, Dave and Fred discuss something. Then, Dave and Fred trade positions. Then, Bob and Gertrude trade positions. Then, Bob and Gertrude discuss something (let's call it Action 59). Then, Bob and Gertrude trade positions. Then, Alice and Claire trade positions. Then, Alice and Claire discuss something (let's call it Action 60). Then, Action 55 repeats. Then, Action 29 repeats. Then, Gertrude and Fred discuss something (let's call it Action 61). Then, Gertrude and Fred trade positions. Then, Dave and Fred trade positions. Then, nothing happens for a while. Then, Fred and Eve trade positions. Then, Action 2 repeats. Then, Action 39 repeats. Then, Claire and Dave discuss something (let's call it Action 62). Then, Claire and Dave trade positions. Then, nothing happens for a while. Then, Fred and Gertrude trade positions. Then, Gertrude and Alice trade positions. Then, Alice and Gertrude trade positions. Then, Gertrude and Fred trade positions. Then, Claire and Dave trade positions. Then, Action 21 repeats. Then, Bob and Dave trade positions. Then, Dave and Claire trade positions. Then, Dave and Alice trade positions. Then, Action 43 repeats. Then, Action 55 repeats. Then, Action 60 repeats. Then, Alice and Claire trade positions. Then, Action 31 repeats. Then, Action 11 repeats. Then, Gertrude and Bob trade positions. Then, Gertrude and Bob discuss something (let's call it Action 63). Then, Gertrude and Bob trade positions. Then, nothing happens for a while. Then, Gertrude and Alice trade positions. Then, Gertrude and Alice discuss something. Then, Gertrude and Alice trade positions. Then, Action 15 repeats. Then, nothing happens for a while. Then, Bob and Claire trade positions. Then, Bob and Claire discuss something. Then, Bob and Claire trade positions. Then, Claire and Alice trade positions. Then, Claire and Alice discuss something. Then, Action 53 repeats. Then, Claire and Fred trade positions. Then, Fred and Claire trade positions. Then, Fred and Bob trade positions. Then, Bob and Gertrude trade positions. Then, Action 7 repeats. Then, Bob and Fred trade positions. Then, Bob and Dave trade positions. Then, Bob and Dave discuss something (let's call it Action 64). Then, Bob and Dave trade positions. Then, nothing happens for a while. Then, Dave and Bob trade positions. Then, Action 22 repeats. Then, Action 21 repeats. Then, Action 33 repeats. Then, Bob and Eve trade positions. Finally, Claire and Alice trade positions. At the end of the match, Gertrude is playing\\nOptions:\\n(A) striker\\n(B) right winger\\n(C) left winger\\n(D) benchwarmer\\n(E) goalkeeper\\n(F) center midfielder\\n(G) cheerleader\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}