{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"07ae8240-ab57-52fd-a9bf-5729c01f2c2c","task_key":"bbeh~5fshuffled~5fobjects--full--07ae8240-ab57-52fd-a9bf-5729c01f2c2c","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are playing a game. At the…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are playing a game. At the start of the game, they are each holding a ball: Alice has a purple ball, Bob has a pink ball, Claire has a orange ball, Dave has a black ball, Eve has a brown ball, Fred has a green ball, and Gertrude has a blue ball.\\nAs the game progresses, pairs of players trade balls. First, Gertrude and Alice swap balls. Then, Bob and Alice swap balls. Then, Alice and Bob swap balls. Then, Dave and Bob swap balls. Then, Dave and Bob discuss something. Then, Dave and Bob swap balls. Then, Gertrude and Eve swap balls. Then, Eve and Dave swap balls. Then, Dave and Eve swap balls. Then, Eve and Gertrude swap balls. Then, Bob and Alice swap balls. Then, Bob and Alice discuss something. Then, Bob and Alice swap balls. Then, Eve and Claire swap balls. Then, Claire and Gertrude swap balls. Then, Gertrude and Claire swap balls. Then, Claire and Eve swap balls. Then, Claire and Gertrude swap balls. Then, Gertrude and Claire swap balls. Then, Claire and Alice swap balls. Then, Claire and Alice discuss something. Then, Claire and Alice swap balls. Then, Alice and Eve swap balls. Then, Eve and Alice swap balls. Then, Fred and Dave swap balls. Then, Dave and Alice swap balls. Then, Alice and Dave swap balls. Then, Dave and Fred swap balls. Then, nothing happens for a while. Then, Bob and Eve swap balls. Then, Eve and Gertrude swap balls. Then, Gertrude and Eve swap balls. Then, Eve and Bob swap balls. Then, Dave and Eve swap balls. Then, Eve and Dave swap balls. Then, nothing happens for a while. Then, Fred and Claire swap balls. Then, Claire and Fred swap balls. Then, Alice and Fred swap balls. Then, Fred and Dave swap balls. Then, Dave and Fred swap balls. Then, Fred and Alice swap balls. Then, nothing happens for a while. Then, Alice and Gertrude swap balls. Then, Gertrude and Bob swap balls. Then, Bob and Gertrude swap balls. Then, Gertrude and Alice swap balls. Then, Fred and Eve swap balls. Then, Fred and Eve discuss something. Then, Fred and Eve swap balls. Then, Claire and Dave swap balls. Then, Claire and Dave discuss something. Then, Claire and Dave swap balls. Then, Fred and Alice swap balls. Then, Alice and Fred swap balls. Then, Alice and Fred swap balls. Then, Fred and Alice swap balls. Then, Dave and Fred swap balls. Then, Dave and Fred discuss something. Then, Dave and Fred swap balls. Then, Gertrude and Fred swap balls. Then, Gertrude and Fred discuss something. Then, Gertrude and Fred swap balls. Then, Eve and Bob swap balls. Then, Eve and Bob discuss something. Then, Eve and Bob swap balls. Then, Gertrude and Alice swap balls. Then, Alice and Gertrude swap balls. Then, Gertrude and Claire swap balls. Then, Gertrude and Claire discuss something. Then, Gertrude and Claire swap balls. Then, Dave and Gertrude swap balls. Then, Gertrude and Claire swap balls. Then, Claire and Gertrude swap balls. Then, Gertrude and Dave swap balls. Then, Claire and Dave swap balls. Then, Dave and Eve swap balls. Then, Eve and Dave swap balls. Then, Dave and Claire swap balls. Then, Eve and Dave swap balls. Then, Dave and Eve swap balls. Then, Bob and Alice swap balls. Then, Alice and Dave swap balls. Then, Dave and Alice swap balls. Then, Alice and Bob swap balls. Then, Dave and Alice swap balls. Then, Alice and Fred swap balls. Then, Fred and Alice swap balls. Then, Alice and Dave swap balls. Then, Dave and Eve swap balls. Then, Eve and Dave swap balls. Then, Eve and Dave swap balls. Then, Eve and Dave discuss something. Then, Eve and Dave swap balls. Then, Gertrude and Bob swap balls. Then, Bob and Dave swap balls. Then, Dave and Bob swap balls. Then, Bob and Gertrude swap balls. Then, Claire and Dave swap balls. Then, Dave and Gertrude swap balls. Then, Gertrude and Dave swap balls. Then, Dave and Claire swap balls. Then, Fred and Bob swap balls. Then, Gertrude and Dave swap balls. Then, Gertrude and Dave discuss something. Then, Gertrude and Dave swap balls. Then, Gertrude and Fred swap balls. Then, Fred and Gertrude swap balls. Then, Gertrude and Claire swap balls. Then, Claire and Gertrude swap balls. Then, Alice and Fred swap balls. Then, Fred and Bob swap balls. Then, Bob and Fred swap balls. Then, Fred and Alice swap balls. Then, Alice and Fred swap balls. Then, Alice and Fred discuss something. Then, Alice and Fred swap balls. Then, Bob and Fred swap balls. Then, Fred and Bob swap balls. Then, Alice and Bob swap balls. Then, Bob and Alice swap balls. Then, Dave and Fred swap balls. Then, Fred and Dave swap balls. Then, Dave and Gertrude swap balls. Then, Dave and Gertrude discuss something. Then, Dave and Gertrude swap balls. Then, Claire and Dave swap balls. Then, Claire and Dave discuss something. Then, Claire and Dave swap balls. Then, Gertrude and Eve swap balls. Then, Gertrude and Eve discuss something. Then, Gertrude and Eve swap balls. Then, Claire and Alice swap balls. Then, Alice and Claire swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, Bob and Claire swap balls. Then, Bob and Claire discuss something. Then, Bob and Claire swap balls. Then, Dave and Bob swap balls. Then, Dave and Bob discuss something. Then, Dave and Bob swap balls. Then, Fred and Alice swap balls. Then, Fred and Alice discuss something. Then, Fred and Alice swap balls. Then, Dave and Bob swap balls. Then, Bob and Alice swap balls. Then, Alice and Bob swap balls. Then, Bob and Dave swap balls. Then, Eve and Gertrude swap balls. Then, Eve and Gertrude discuss something. Then, Eve and Gertrude swap balls. Then, Alice and Fred swap balls. Then, Alice and Fred discuss something. Then, Alice and Fred swap balls. Then, Eve and Claire swap balls. Then, Claire and Eve swap balls. Then, nothing happens for a while. Then, Dave and Alice swap balls. Then, Alice and Eve swap balls. Then, Eve and Alice swap balls. Then, Alice and Dave swap balls. Then, Dave and Alice swap balls. Then, Alice and Claire swap balls. Then, Claire and Alice swap balls. Then, Alice and Dave swap balls. Then, Alice and Eve swap balls. Then, Alice and Eve discuss something. Then, Alice and Eve swap balls. Then, Eve and Claire swap balls. Then, Claire and Fred swap balls. Then, Fred and Claire swap balls. Then, Claire and Eve swap balls. Then, nothing happens for a while. Then, Gertrude and Eve swap balls. Then, Eve and Gertrude swap balls. Then, Alice and Bob swap balls. Then, Bob and Alice swap balls. Then, Eve and Bob swap balls. Then, Bob and Dave swap balls. Then, Dave and Bob swap balls. Then, Bob and Eve swap balls. Then, Claire and Alice swap balls. Then, Alice and Bob swap balls. Then, Bob and Alice swap balls. Then, Alice and Claire swap balls. Then, Fred and Dave swap balls. Then, Dave and Fred swap balls. Then, Dave and Claire swap balls. Then, Claire and Bob swap balls. Then, Bob and Claire swap balls. Then, Claire and Dave swap balls. Then, Gertrude and Eve swap balls. Then, Eve and Alice swap balls. Then, Alice and Eve swap balls. Then, Eve and Gertrude swap balls. Then, nothing happens for a while. Then, Bob and Alice swap balls. Then, Bob and Alice discuss something. Then, Bob and Alice swap balls. Then, nothing happens for a while. Then, Claire and Bob swap balls. Then, Bob and Dave swap balls. Then, Dave and Bob swap balls. Then, Bob and Claire swap balls. Then, Gertrude and Bob swap balls. Then, Gertrude and Bob discuss something. Then, Gertrude and Bob swap balls. Then, Claire and Dave swap balls. Then, Dave and Claire swap balls. Then, Bob and Eve swap balls. Then, Eve and Bob swap balls. Then, Fred and Claire swap balls. Then, Fred and Alice swap balls. Then, Alice and Gertrude swap balls. Then, Gertrude and Alice swap balls. Then, Alice and Fred swap balls. Then, Eve and Bob swap balls. Then, Bob and Gertrude swap balls. Then, Gertrude and Bob swap balls. Then, Bob and Eve swap balls. Then, Claire and Bob swap balls. Then, Bob and Claire swap balls. Then, Fred and Gertrude swap balls. Then, Gertrude and Dave swap balls. Then, Dave and Gertrude swap balls. Then, Gertrude and Fred swap balls. Then, Bob and Dave swap balls. Then, Dave and Bob swap balls. Then, Claire and Fred swap balls. Then, Fred and Claire swap balls. Then, Gertrude and Fred swap balls. Then, Fred and Eve swap balls. Then, Eve and Fred swap balls. Then, Fred and Gertrude swap balls. Then, nothing happens for a while. Then, Alice and Dave swap balls. Then, Alice and Dave discuss something. Then, Alice and Dave swap balls. Then, Alice and Claire swap balls. Then, Alice and Claire discuss something. Then, Alice and Claire swap balls. Then, Fred and Dave swap balls. Then, Dave and Eve swap balls. Then, Eve and Dave swap balls. Then, Dave and Fred swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Eve swap balls. Then, Eve and Fred swap balls. Then, Fred and Eve swap balls. Then, Eve and Dave swap balls. Then, Bob and Claire swap balls. Then, Bob and Claire discuss something. Then, Bob and Claire swap balls. Then, Fred and Dave swap balls. Then, Dave and Fred swap balls. Then, Bob and Gertrude swap balls. Then, Gertrude and Eve swap balls. Then, Eve and Gertrude swap balls. Then, Gertrude and Bob swap balls. Then, Gertrude and Bob swap balls. Then, Bob and Gertrude swap balls. Then, nothing happens for a while. Then, Gertrude and Claire swap balls. Then, Claire and Gertrude swap balls. Then, Bob and Dave swap balls. Then, Bob and Dave discuss something. Then, Bob and Dave swap balls. Then, nothing happens for a while. Then, Dave and Bob swap balls. Then, Dave and Bob discuss something. Then, Dave and Bob swap balls. Then, Bob and Fred swap balls. Then, Fred and Bob swap balls. Then, Alice and Claire swap balls. Then, Claire and Eve swap balls. Then, Eve and Claire swap balls. Then, Claire and Alice swap balls. Then, Eve and Dave swap balls. Then, Alice and Bob swap balls. Then, Alice and Bob discuss something. Then, Alice and Bob swap balls. Then, nothing happens for a while. Then, Gertrude and Alice swap balls. Then, Alice and Claire swap balls. Then, Claire and Alice swap balls. Then, Alice and Gertrude swap balls. Then, Dave and Eve swap balls. Then, Dave and Eve discuss something. Then, Dave and Eve swap balls. Then, Eve and Bob swap balls. Then, Bob and Eve swap balls. Then, nothing happens for a while. Then, Claire and Fred swap balls. Then, Fred and Claire swap balls. Then, nothing happens for a while. Then, Gertrude and Dave swap balls. Then, Dave and Gertrude swap balls. Then, Claire and Eve swap balls. Then, Claire and Eve discuss something. Then, Claire and Eve swap balls. Then, Eve and Claire swap balls. Then, Claire and Dave swap balls. Then, Dave and Claire swap balls. Then, Claire and Eve swap balls. Then, nothing happens for a while. Then, Bob and Alice swap balls. Then, Alice and Bob swap balls. Then, Fred and Eve swap balls. Then, Eve and Fred swap balls. Then, nothing happens for a while. Then, Claire and Eve swap balls. Then, Eve and Dave swap balls. Then, Dave and Eve swap balls. Then, Eve and Claire swap balls. Then, nothing happens for a while. Then, Eve and Bob swap balls. Then, Bob and Eve swap balls. Then, Alice and Gertrude swap balls. Then, Alice and Gertrude discuss something. Then, Alice and Gertrude swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Claire swap balls. Then, Claire and Dave swap balls. Then, Bob and Alice swap balls. Then, Alice and Fred swap balls. Then, Fred and Alice swap balls. Then, Alice and Bob swap balls. Then, Alice and Dave swap balls. Then, Dave and Fred swap balls. Then, Fred and Dave swap balls. Then, Dave and Alice swap balls. Then, nothing happens for a while. Then, Fred and Bob swap balls. Then, Bob and Fred swap balls. Then, Dave and Claire swap balls. Then, Dave and Claire discuss something. Then, Dave and Claire swap balls. Then, Alice and Eve swap balls. Then, Alice and Claire swap balls. Finally, Claire and Fred swap balls. At the end of the game, Eve has the\\nOptions:\\n(A) purple ball\\n(B) pink ball\\n(C) orange ball\\n(D) black ball\\n(E) brown ball\\n(F) green ball\\n(G) blue ball\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}