{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"44b137b3-6279-5837-88a2-ad1086030602","task_key":"bbeh~5fshuffled~5fobjects--full--44b137b3-6279-5837-88a2-ad1086030602","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are on the same team in a…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are on the same team in a soccer match. At the start of the match, they are each assigned to a position: Alice is playing benchwarmer, Bob is playing goalkeeper, Claire is playing striker, Dave is playing right midfielder, Eve is playing center midfielder, Fred is playing cheerleader, and Gertrude is playing left midfielder.\\nAs the game progresses, pairs of players occasionally swap positions. First, Dave and Claire trade positions. Then, nothing happens for a while. Then, Eve and Gertrude trade positions. Then, Eve and Gertrude discuss something. Then, Eve and Gertrude trade positions. Then, nothing happens for a while. Then, Claire and Dave trade positions. Then, Dave and Alice trade positions. Then, Alice and Dave trade positions. Then, Dave and Claire trade positions. Then, nothing happens for a while. Then, nothing happens for a while. Then, Eve and Gertrude trade positions. Then, Eve and Gertrude discuss something. Then, Eve and Gertrude trade positions. Then, Dave and Claire trade positions. Then, Dave and Claire discuss something. Then, Dave and Claire trade positions. Then, Eve and Dave trade positions. Then, Dave and Alice trade positions. Then, Alice and Dave trade positions. Then, Dave and Eve trade positions. Then, nothing happens for a while. Then, Eve and Alice trade positions. Then, Alice and Eve trade positions. Then, nothing happens for a while. Then, Bob and Fred trade positions. Then, Fred and Alice trade positions. Then, Alice and Fred trade positions. Then, Fred and Bob trade positions. Then, Eve and Alice trade positions. Then, Alice and Claire trade positions. Then, Claire and Alice trade positions. Then, Alice and Eve trade positions. Then, Dave and Alice trade positions. Then, Alice and Dave trade positions. Then, Gertrude and Bob trade positions. Then, Gertrude and Bob discuss something. Then, Gertrude and Bob trade positions. Then, Claire and Alice trade positions. Then, Claire and Alice discuss something. Then, Claire and Alice trade positions. Then, Gertrude and Dave trade positions. Then, Gertrude and Dave discuss something. Then, Gertrude and Dave trade positions. Then, nothing happens for a while. Then, Gertrude and Dave trade positions. Then, Dave and Gertrude trade positions. Then, Fred and Alice trade positions. Then, Fred and Alice discuss something. Then, Fred and Alice trade positions. Then, Bob and Gertrude trade positions. Then, Bob and Gertrude discuss something. Then, Bob and Gertrude trade positions. Then, Fred and Eve trade positions. Then, Eve and Dave trade positions. Then, Dave and Eve trade positions. Then, Eve and Fred trade positions. Then, Bob and Fred trade positions. Then, Fred and Bob trade positions. Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Claire trade positions. Then, Claire and Alice trade positions. Then, Eve and Fred trade positions. Then, Eve and Fred discuss something. Then, Eve and Fred trade positions. Then, nothing happens for a while. Then, Eve and Dave trade positions. Then, Dave and Claire trade positions. Then, Claire and Dave trade positions. Then, Dave and Eve trade positions. Then, Dave and Eve trade positions. Then, Eve and Dave trade positions. Then, nothing happens for a while. Then, Alice and Fred trade positions. Then, Fred and Gertrude trade positions. Then, Gertrude and Fred trade positions. Then, Fred and Alice trade positions. Then, Fred and Gertrude trade positions. Then, Fred and Gertrude discuss something. Then, Fred and Gertrude trade positions. Then, Dave and Gertrude trade positions. Then, Dave and Gertrude discuss something. Then, Dave and Gertrude trade positions. Then, Claire and Bob trade positions. Then, Claire and Bob discuss something. Then, Claire and Bob trade positions. Then, nothing happens for a while. Then, Claire and Fred trade positions. Then, Fred and Claire trade positions. Then, Fred and Dave trade positions. Then, Dave and Fred trade positions. Then, nothing happens for a while. Then, Claire and Fred trade positions. Then, Fred and Gertrude trade positions. Then, Gertrude and Fred trade positions. Then, Fred and Claire trade positions. Then, Fred and Eve trade positions. Then, Fred and Eve discuss something. Then, Fred and Eve trade positions. Then, nothing happens for a while. Then, nothing happens for a while. Then, Gertrude and Alice trade positions. Then, Gertrude and Alice discuss something. Then, Gertrude and Alice trade positions. Then, Dave and Bob trade positions. Then, Bob and Gertrude trade positions. Then, Gertrude and Bob trade positions. Then, Bob and Dave trade positions. Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Fred trade positions. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Claire and Alice trade positions. Then, Alice and Claire trade positions. Then, Alice and Eve trade positions. Then, Eve and Gertrude trade positions. Then, Gertrude and Eve trade positions. Then, Eve and Alice trade positions. Then, nothing happens for a while. Then, Bob and Alice trade positions. Then, Alice and Bob trade positions. Then, nothing happens for a while. Then, Gertrude and Alice trade positions. Then, Alice and Gertrude trade positions. Then, Fred and Alice trade positions. Then, Alice and Eve trade positions. Then, Eve and Alice trade positions. Then, Alice and Fred trade positions. Then, Claire and Dave trade positions. Then, Dave and Bob trade positions. Then, Bob and Dave trade positions. Then, Dave and Claire trade positions. Then, Fred and Eve trade positions. Then, Eve and Fred trade positions. Then, Claire and Dave trade positions. Then, Dave and Claire trade positions. Then, nothing happens for a while. Then, Gertrude and Bob trade positions. Then, Bob and Eve trade positions. Then, Eve and Bob trade positions. Then, Bob and Gertrude trade positions. Then, Bob and Claire trade positions. Then, Claire and Alice trade positions. Then, Alice and Claire trade positions. Then, Claire and Bob trade positions. Then, Alice and Bob trade positions. Then, Bob and Claire trade positions. Then, Claire and Bob trade positions. Then, Bob and Alice trade positions. Then, Dave and Eve trade positions. Then, Dave and Eve discuss something. Then, Dave and Eve trade positions. Then, nothing happens for a while. Then, Gertrude and Bob trade positions. Then, Bob and Gertrude trade positions. Then, nothing happens for a while. Then, Bob and Gertrude trade positions. Then, Gertrude and Claire trade positions. Then, Claire and Gertrude trade positions. Then, Gertrude and Bob trade positions. Then, Claire and Bob trade positions. Then, Bob and Claire trade positions. Then, nothing happens for a while. Then, Dave and Alice trade positions. Then, Alice and Claire trade positions. Then, Claire and Alice trade positions. Then, Alice and Dave trade positions. Then, Alice and Gertrude trade positions. Then, Gertrude and Alice trade positions. Then, nothing happens for a while. Then, Eve and Alice trade positions. Then, Alice and Gertrude trade positions. Then, Gertrude and Alice trade positions. Then, Alice and Eve trade positions. Then, Claire and Gertrude trade positions. Then, Claire and Gertrude discuss something. Then, Claire and Gertrude trade positions. Then, Bob and Gertrude trade positions. Then, Gertrude and Dave trade positions. Then, Dave and Gertrude trade positions. Then, Gertrude and Bob trade positions. Then, Claire and Fred trade positions. Then, Gertrude and Claire trade positions. Then, Claire and Eve trade positions. Then, Eve and Claire trade positions. Then, Claire and Gertrude trade positions. Then, nothing happens for a while. Then, nothing happens for a while. Then, Fred and Eve trade positions. Then, Eve and Fred trade positions. Then, Alice and Gertrude trade positions. Then, Gertrude and Dave trade positions. Then, Dave and Gertrude trade positions. Then, Gertrude and Alice trade positions. Then, nothing happens for a while. Then, Gertrude and Bob trade positions. Then, Gertrude and Bob discuss something. Then, Gertrude and Bob trade positions. Then, Gertrude and Fred trade positions. Then, Fred and Dave trade positions. Then, Dave and Fred trade positions. Then, Fred and Gertrude trade positions. Then, Bob and Fred trade positions. Then, Fred and Bob trade positions. Then, Fred and Dave trade positions. Then, Dave and Fred trade positions. Then, Dave and Alice trade positions. Then, Alice and Fred trade positions. Then, Fred and Alice trade positions. Then, Alice and Dave trade positions. Then, Bob and Gertrude trade positions. Then, Bob and Gertrude discuss something. Then, Bob and Gertrude trade positions. Then, Claire and Bob trade positions. Then, Bob and Claire trade positions. Then, nothing happens for a while. Then, Bob and Alice trade positions. Then, Alice and Bob trade positions. Then, Bob and Fred trade positions. Then, Fred and Bob trade positions. Then, nothing happens for a while. Then, Gertrude and Alice trade positions. Then, Alice and Bob trade positions. Then, Bob and Alice trade positions. Then, Alice and Gertrude trade positions. Then, Alice and Claire trade positions. Then, Claire and Alice trade positions. Then, nothing happens for a while. Then, Gertrude and Bob trade positions. Then, Gertrude and Bob discuss something. Then, Gertrude and Bob trade positions. Then, Eve and Alice trade positions. Then, Alice and Eve trade positions. Then, Claire and Fred trade positions. Then, Fred and Dave trade positions. Then, Dave and Fred trade positions. Then, Fred and Claire trade positions. Then, Claire and Dave trade positions. Then, Claire and Dave discuss something. Then, Claire and Dave trade positions. Then, Alice and Eve trade positions. Then, Alice and Eve discuss something. Then, Alice and Eve trade positions. Then, Claire and Bob trade positions. Then, Bob and Claire trade positions. Then, Fred and Gertrude trade positions. Then, Gertrude and Claire trade positions. Then, Claire and Gertrude trade positions. Then, Gertrude and Fred trade positions. Then, Fred and Dave trade positions. Then, Dave and Fred trade positions. Then, Gertrude and Claire trade positions. Then, Claire and Gertrude trade positions. Then, Bob and Dave trade positions. Then, Dave and Claire trade positions. Then, Claire and Dave trade positions. Then, Dave and Bob trade positions. Then, Eve and Claire trade positions. Then, Eve and Claire discuss something. Then, Eve and Claire trade positions. Then, nothing happens for a while. Then, Bob and Claire trade positions. Then, Claire and Bob trade positions. Then, Fred and Alice trade positions. Then, Fred and Alice discuss something. Then, Fred and Alice trade positions. Then, Claire and Bob trade positions. Then, Claire and Bob discuss something. Then, Claire and Bob trade positions. Then, Eve and Alice trade positions. Then, Alice and Bob trade positions. Then, Bob and Alice trade positions. Then, Alice and Eve trade positions. Then, Alice and Claire trade positions. Then, Claire and Alice trade positions. Then, Dave and Eve trade positions. Then, Dave and Eve discuss something. Then, Dave and Eve trade positions. Then, Gertrude and Eve trade positions. Then, Eve and Claire trade positions. Then, Claire and Eve trade positions. Then, Eve and Gertrude trade positions. Then, nothing happens for a while. Then, Dave and Claire trade positions. Then, Claire and Dave trade positions. Then, Alice and Claire trade positions. Then, Alice and Claire discuss something. Then, Alice and Claire trade positions. Then, Eve and Gertrude trade positions. Then, Claire and Bob trade positions. Then, Bob and Alice trade positions. Then, Alice and Bob trade positions. Then, Bob and Claire trade positions. Then, nothing happens for a while. Then, Fred and Gertrude trade positions. Then, Fred and Gertrude discuss something. Then, Fred and Gertrude trade positions. Then, Eve and Claire trade positions. Then, Claire and Eve trade positions. Then, Claire and Alice trade positions. Then, Alice and Fred trade positions. Then, Fred and Alice trade positions. Then, Alice and Claire trade positions. Then, Claire and Alice trade positions. Then, Claire and Alice discuss something. Then, Claire and Alice trade positions. Then, Claire and Bob trade positions. Then, Bob and Fred trade positions. Then, Fred and Bob trade positions. Then, Bob and Claire trade positions. Then, nothing happens for a while. Then, Claire and Bob trade positions. Then, Bob and Eve trade positions. Then, Eve and Bob trade positions. Then, Bob and Claire trade positions. Then, Dave and Gertrude trade positions. Then, Dave and Gertrude discuss something. Then, Dave and Gertrude trade positions. Then, Eve and Gertrude trade positions. Then, Gertrude and Eve trade positions. Then, Bob and Alice trade positions. Then, Alice and Bob trade positions. Then, Dave and Gertrude trade positions. Then, Dave and Gertrude discuss something. Then, Dave and Gertrude trade positions. Then, nothing happens for a while. Then, nothing happens for a while. Then, Claire and Dave trade positions. Then, Dave and Alice trade positions. Then, Alice and Dave trade positions. Then, Dave and Claire trade positions. Then, nothing happens for a while. Then, Alice and Eve trade positions. Then, Eve and Alice trade positions. Then, nothing happens for a while. Then, Eve and Dave trade positions. Then, Dave and Gertrude trade positions. Then, Gertrude and Dave trade positions. Then, Dave and Eve trade positions. Then, nothing happens for a while. Then, nothing happens for a while. Then, Fred and Gertrude trade positions. Then, Gertrude and Dave trade positions. Then, Dave and Gertrude trade positions. Then, Gertrude and Fred trade positions. Then, Fred and Dave trade positions. Then, Fred and Dave discuss something. Then, Fred and Dave trade positions. Then, Claire and Eve trade positions. Then, Eve and Claire trade positions. Then, Dave and Fred trade positions. Then, Fred and Dave trade positions. Then, Dave and Eve trade positions. Then, Eve and Alice trade positions. Then, Alice and Eve trade positions. Then, Eve and Dave trade positions. Then, Bob and Claire trade positions. Then, Bob and Claire discuss something. Then, Bob and Claire trade positions. Then, nothing happens for a while. Then, Dave and Alice trade positions. Then, Alice and Dave trade positions. Then, Alice and Fred trade positions. Then, Alice and Fred discuss something. Then, Alice and Fred trade positions. Then, Gertrude and Alice trade positions. Then, Alice and Gertrude trade positions. Then, nothing happens for a while. Then, Claire and Gertrude trade positions. Then, Claire and Gertrude discuss something. Then, Claire and Gertrude trade positions. Then, Alice and Dave trade positions. Then, Dave and Fred trade positions. Then, Fred and Dave trade positions. Then, Dave and Alice trade positions. Then, Eve and Bob trade positions. Then, Eve and Bob discuss something. Then, Eve and Bob trade positions. Then, Dave and Bob trade positions. Then, Bob and Fred trade positions. Then, Fred and Bob trade positions. Then, Bob and Dave trade positions. Then, Bob and Fred trade positions. Then, Fred and Dave trade positions. Then, Dave and Fred trade positions. Then, Fred and Bob trade positions. Then, Gertrude and Fred trade positions. Then, Gertrude and Fred discuss something. Then, Gertrude and Fred trade positions. Then, Gertrude and Alice trade positions. Then, Alice and Gertrude trade positions. Then, Fred and Claire trade positions. Then, Claire and Fred trade positions. Then, Fred and Bob trade positions. Then, Bob and Dave trade positions. Then, Dave and Bob trade positions. Then, Bob and Fred trade positions. Then, Eve and Fred trade positions. Then, Fred and Claire trade positions. Then, Claire and Fred trade positions. Then, Fred and Eve trade positions. Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Alice trade positions. Then, Dave and Alice discuss something. Then, Dave and Alice trade positions. Then, Gertrude and Dave trade positions. Then, Dave and Eve trade positions. Then, Eve and Dave trade positions. Then, Dave and Gertrude trade positions. Then, Alice and Fred trade positions. Then, Dave and Fred trade positions. Finally, Dave and Bob trade positions. At the end of the match, Gertrude is playing\\nOptions:\\n(A) benchwarmer\\n(B) goalkeeper\\n(C) striker\\n(D) right midfielder\\n(E) center midfielder\\n(F) cheerleader\\n(G) left midfielder\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}