{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9a6f7fb6-ec07-5bfc-9f4a-943b7a97884a","task_key":"bbeh~5fshuffled~5fobjects--full--9a6f7fb6-ec07-5bfc-9f4a-943b7a97884a","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are on the same team in a…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are on the same team in a soccer match. At the start of the match, they are each assigned to a position: Alice is playing center midfielder, Bob is playing striker, Claire is playing left winger, Dave is playing cheerleader, Eve is playing fullback, Fred is playing right winger, and Gertrude is playing benchwarmer.\\nAs the game progresses, pairs of players occasionally swap positions. First, Alice and Claire trade positions. Then, Claire and Gertrude trade positions (let's call it Action 1). Then, Gertrude and Claire trade positions (let's call it Action 2). Then, Gertrude and Dave trade positions (let's call it Action 3). Then, Dave and Gertrude trade positions (let's call it Action 4). Then, Eve and Dave trade positions (let's call it Action 5). Then, Action 4 repeats. Then, Gertrude and Dave trade positions. Then, Dave and Eve trade positions (let's call it Action 6). Then, Bob and Alice trade positions (let's call it Action 7). Then, Alice and Bob trade positions (let's call it Action 8). Then, nothing happens for a while (let's call it Action 9). Then, Gertrude and Claire trade positions. Then, Gertrude and Claire discuss something (let's call it Action 10). Then, Gertrude and Claire trade positions. Then, Fred and Eve trade positions (let's call it Action 11). Then, Fred and Eve discuss something (let's call it Action 12). Then, Fred and Eve trade positions. Then, nothing happens for a while. Then, Fred and Claire trade positions (let's call it Action 13). Then, Claire and Fred trade positions (let's call it Action 14). Then, Action 9 repeats. Then, Eve and Bob trade positions (let's call it Action 15). Then, Bob and Eve trade positions (let's call it Action 16). Then, nothing happens for a while. Then, nothing happens for a while. Then, Bob and Alice trade positions. Then, Bob and Alice discuss something (let's call it Action 17). Then, Bob and Alice trade positions. Then, Eve and Claire trade positions (let's call it Action 18). Then, Eve and Claire discuss something (let's call it Action 19). Then, Eve and Claire trade positions. Then, nothing happens for a while. Then, Eve and Dave trade positions. Then, Dave and Eve trade positions. Then, Gertrude and Fred trade positions (let's call it Action 20). Then, Fred and Gertrude trade positions (let's call it Action 21). Then, Eve and Bob trade positions. Then, Bob and Eve trade positions. Then, Fred and Bob trade positions (let's call it Action 22). Then, Fred and Bob discuss something (let's call it Action 23). Then, Fred and Bob trade positions. Then, Dave and Alice trade positions (let's call it Action 24). Then, Alice and Bob trade positions. Then, Bob and Alice trade positions. Then, Alice and Dave trade positions (let's call it Action 25). Then, Eve and Gertrude trade positions (let's call it Action 26). Then, Gertrude and Eve trade positions (let's call it Action 27). Then, Gertrude and Alice trade positions (let's call it Action 28). Then, Gertrude and Alice discuss something (let's call it Action 29). Then, Gertrude and Alice trade positions. Then, Action 15 repeats. Then, Bob and Fred trade positions (let's call it Action 30). Then, Fred and Bob trade positions. Then, Action 16 repeats. Then, Fred and Alice trade positions (let's call it Action 31). Then, Fred and Alice discuss something (let's call it Action 32). Then, Fred and Alice trade positions. Then, Action 20 repeats. Then, Fred and Eve trade positions. Then, Eve and Fred trade positions (let's call it Action 33). Then, Action 21 repeats. Then, Eve and Bob trade positions. Then, Bob and Eve trade positions. Then, nothing happens for a while. Then, Dave and Bob trade positions (let's call it Action 34). Then, Dave and Bob discuss something (let's call it Action 35). Then, Action 34 repeats. Then, Claire and Fred trade positions. Then, Claire and Fred discuss something (let's call it Action 36). Then, Action 14 repeats. Then, Eve and Dave trade positions. Then, Eve and Dave discuss something (let's call it Action 37). Then, Action 5 repeats. Then, Action 21 repeats. Then, Gertrude and Fred trade positions. Then, Alice and Fred trade positions (let's call it Action 38). Then, Fred and Claire trade positions. Then, Claire and Fred trade positions. Then, Fred and Alice trade positions. Then, Dave and Bob trade positions. Then, Bob and Dave trade positions (let's call it Action 39). Then, Claire and Bob trade positions (let's call it Action 40). Then, Bob and Claire trade positions (let's call it Action 41). Then, Gertrude and Bob trade positions (let's call it Action 42). Then, nothing happens for a while. Then, Bob and Gertrude trade positions (let's call it Action 43). Then, Bob and Gertrude discuss something (let's call it Action 44). Then, Bob and Gertrude trade positions. Then, Action 33 repeats. Then, Eve and Fred discuss something (let's call it Action 45). Then, Eve and Fred trade positions. Then, Action 25 repeats. Then, Dave and Alice trade positions. Then, nothing happens for a while. Then, Fred and Eve trade positions. Then, Eve and Claire trade positions. Then, Claire and Eve trade positions (let's call it Action 46). Then, Eve and Fred trade positions. Then, Action 18 repeats. Then, Eve and Claire discuss something. Then, Eve and Claire trade positions. Then, Action 39 repeats. Then, Bob and Dave discuss something (let's call it Action 47). Then, Bob and Dave trade positions. Then, Dave and Bob trade positions. Then, Bob and Eve trade positions. Then, Eve and Bob trade positions. Then, Bob and Dave trade positions. Then, Gertrude and Dave trade positions. Then, Dave and Gertrude trade positions. Then, Gertrude and Dave trade positions. Then, Gertrude and Dave discuss something (let's call it Action 48). Then, Action 3 repeats. Then, Action 13 repeats. Then, Claire and Fred trade positions. Then, nothing happens for a while. Then, Dave and Gertrude trade positions. Then, Gertrude and Eve trade positions. Then, Eve and Gertrude trade positions. Then, Gertrude and Dave trade positions. Then, Eve and Dave trade positions. Then, Action 37 repeats. Then, Eve and Dave trade positions. Then, Alice and Claire trade positions (let's call it Action 49). Then, Action 1 repeats. Then, Gertrude and Claire trade positions. Then, Claire and Alice trade positions (let's call it Action 50). Then, Bob and Alice trade positions. Then, Action 8 repeats. Then, Action 27 repeats. Then, Eve and Gertrude trade positions. Then, nothing happens for a while. Then, Gertrude and Fred trade positions. Then, Fred and Claire trade positions. Then, Action 14 repeats. Then, Fred and Gertrude trade positions. Then, Alice and Claire trade positions. Then, Alice and Claire discuss something (let's call it Action 51). Then, Action 49 repeats. Then, Dave and Gertrude trade positions. Then, Gertrude and Claire trade positions. Then, Action 1 repeats. Then, Gertrude and Dave trade positions. Then, Fred and Bob trade positions. Then, Bob and Fred trade positions. Then, Eve and Bob trade positions. Then, Eve and Bob discuss something (let's call it Action 52). Then, Action 15 repeats. Then, Action 30 repeats. Then, Action 21 repeats. Then, Action 20 repeats. Then, Fred and Bob trade positions. Then, Gertrude and Claire trade positions. Then, Claire and Gertrude trade positions. Then, Bob and Alice trade positions. Then, Bob and Alice discuss something. Then, Bob and Alice trade positions. Then, nothing happens for a while. Then, Action 7 repeats. Then, Alice and Eve trade positions (let's call it Action 53). Then, Eve and Alice trade positions (let's call it Action 54). Then, Alice and Bob trade positions. Then, Eve and Gertrude trade positions. Then, Action 20 repeats. Then, Action 21 repeats. Then, Gertrude and Eve trade positions. Then, Eve and Alice trade positions. Then, Action 24 repeats. Then, Action 25 repeats. Then, Action 22 repeats. Then, Bob and Claire trade positions. Then, Claire and Bob trade positions. Then, Bob and Fred trade positions. Then, Action 20 repeats. Then, Fred and Dave trade positions (let's call it Action 55). Then, Dave and Fred trade positions (let's call it Action 56). Then, Action 21 repeats. Then, Gertrude and Fred trade positions. Then, Fred and Alice trade positions. Then, Alice and Fred trade positions. Then, Fred and Gertrude trade positions. Then, Claire and Dave trade positions (let's call it Action 57). Then, Dave and Claire trade positions (let's call it Action 58). Then, nothing happens for a while. Then, nothing happens for a while. Then, Claire and Fred trade positions. Then, Action 36 repeats. Then, Claire and Fred trade positions. Then, Dave and Gertrude trade positions. Then, Action 27 repeats. Then, Eve and Gertrude trade positions. Then, Action 3 repeats. Then, nothing happens for a while. Then, nothing happens for a while. Then, Bob and Alice trade positions. Then, Bob and Alice discuss something. Then, Bob and Alice trade positions. Then, Eve and Fred trade positions. Then, Fred and Bob trade positions. Then, Bob and Fred trade positions. Then, Action 11 repeats. Then, Alice and Bob trade positions. Then, Action 43 repeats. Then, Gertrude and Bob trade positions. Then, Bob and Alice trade positions. Then, Action 16 repeats. Then, Eve and Fred trade positions. Then, Action 11 repeats. Then, Eve and Bob trade positions. Then, Eve and Bob trade positions. Then, Bob and Alice trade positions. Then, Alice and Bob trade positions. Then, Bob and Eve trade positions. Then, nothing happens for a while. Then, Eve and Claire trade positions. Then, Eve and Claire discuss something. Then, Action 18 repeats. Then, Bob and Fred trade positions. Then, Fred and Bob trade positions. Then, Action 46 repeats. Then, Claire and Eve discuss something (let's call it Action 59). Then, Action 46 repeats. Then, Action 9 repeats. Then, Eve and Alice trade positions. Then, Alice and Eve trade positions. Then, Action 1 repeats. Then, Gertrude and Fred trade positions. Then, Action 21 repeats. Then, Gertrude and Claire trade positions. Then, Dave and Alice trade positions. Then, Action 53 repeats. Then, Eve and Alice trade positions. Then, Action 25 repeats. Then, Fred and Dave trade positions. Then, Dave and Claire trade positions. Then, Claire and Dave trade positions. Then, Action 56 repeats. Then, nothing happens for a while. Then, Claire and Bob trade positions. Then, Bob and Claire trade positions. Then, Alice and Gertrude trade positions (let's call it Action 60). Then, Alice and Gertrude discuss something (let's call it Action 61). Then, Alice and Gertrude trade positions. Then, Eve and Claire trade positions. Then, Claire and Dave trade positions. Then, Dave and Claire trade positions. Then, Claire and Eve trade positions. Then, Claire and Eve trade positions. Then, Claire and Eve discuss something. Then, Action 46 repeats. Then, Action 9 repeats. Then, Gertrude and Claire trade positions. Then, Action 50 repeats. Then, Action 49 repeats. Then, Claire and Gertrude trade positions. Then, Fred and Dave trade positions. Then, Dave and Fred trade positions. Then, nothing happens for a while. Then, Action 9 repeats. Then, Action 9 repeats. Then, Eve and Gertrude trade positions. Then, Eve and Gertrude discuss something (let's call it Action 62). Then, Eve and Gertrude trade positions. Then, nothing happens for a while. Then, nothing happens for a while. Then, Claire and Fred trade positions. Then, Bob and Gertrude trade positions. Then, Action 44 repeats. Then, Bob and Gertrude trade positions. Then, Dave and Claire trade positions. Then, Action 1 repeats. Then, Action 2 repeats. Then, Action 57 repeats. Then, Dave and Eve trade positions. Then, Eve and Dave trade positions. Then, Dave and Fred trade positions. Then, Fred and Bob trade positions. Then, Bob and Fred trade positions. Then, Fred and Dave trade positions. Then, Fred and Dave trade positions. Then, Dave and Fred trade positions. Then, Gertrude and Eve trade positions. Then, Eve and Gertrude trade positions. Then, Action 24 repeats. Then, Action 25 repeats. Then, Action 9 repeats. Then, nothing happens for a while. Then, Action 39 repeats. Then, Dave and Claire trade positions. Then, Claire and Dave trade positions. Then, Dave and Bob trade positions. Then, Alice and Eve trade positions. Then, Eve and Alice trade positions. Then, nothing happens for a while. Then, nothing happens for a while. Then, Eve and Claire trade positions. Then, Claire and Bob trade positions. Then, Bob and Claire trade positions. Then, Claire and Eve trade positions. Then, Action 2 repeats. Then, Claire and Gertrude trade positions. Then, Alice and Gertrude trade positions. Then, Gertrude and Alice trade positions. Then, Action 60 repeats. Then, Action 61 repeats. Then, Action 60 repeats. Then, Action 9 repeats. Then, Alice and Gertrude trade positions. Then, Gertrude and Claire trade positions. Then, Action 1 repeats. Then, Gertrude and Alice trade positions. Then, Fred and Bob trade positions. Then, Fred and Bob discuss something. Then, Fred and Bob trade positions. Then, Fred and Claire trade positions. Then, Fred and Claire discuss something (let's call it Action 63). Then, Action 13 repeats. Then, Action 53 repeats. Then, Eve and Alice trade positions. Then, Bob and Alice trade positions. Then, Action 17 repeats. Then, Bob and Alice trade positions. Then, nothing happens for a while. Then, Gertrude and Eve trade positions. Then, Eve and Claire trade positions. Then, Claire and Eve trade positions. Then, Eve and Gertrude trade positions. Then, Action 14 repeats. Then, Claire and Fred discuss something. Then, Claire and Fred trade positions. Then, nothing happens for a while. Then, Fred and Bob trade positions. Then, Bob and Dave trade positions. Then, Dave and Bob trade positions. Then, Bob and Fred trade positions. Then, Action 24 repeats. Then, Alice and Bob trade positions. Then, Bob and Alice trade positions. Then, Action 25 repeats. Then, Gertrude and Dave trade positions. Then, Eve and Fred trade positions. Finally, Fred and Bob trade positions. At the end of the match, Dave is playing\\nOptions:\\n(A) center midfielder\\n(B) striker\\n(C) left winger\\n(D) cheerleader\\n(E) fullback\\n(F) right winger\\n(G) benchwarmer\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}