{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7e46f5de-45ed-5a82-b26a-609a6fd313ee","task_key":"bbeh~5fshuffled~5fobjects--full--7e46f5de-45ed-5a82-b26a-609a6fd313ee","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are playing a game. At the…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are playing a game. At the start of the game, they are each holding a ball: Alice has a white ball, Bob has a brown ball, Claire has a blue ball, Dave has a pink ball, Eve has a purple ball, Fred has a black ball, and Gertrude has a yellow ball.\\nAs the game progresses, pairs of players trade balls. First, Fred and Dave swap balls. Then, Claire and Fred swap balls. Then, Fred and Alice swap balls. Then, Alice and Fred swap balls. Then, Fred and Claire swap balls. Then, Eve and Fred swap balls. Then, Eve and Fred discuss something. Then, Eve and Fred swap balls. Then, Eve and Fred swap balls. Then, Fred and Alice swap balls. Then, Alice and Fred swap balls. Then, Fred and Eve swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, Eve and Alice swap balls. Then, Alice and Eve swap balls. Then, Fred and Gertrude swap balls. Then, Fred and Gertrude discuss something. Then, Fred and Gertrude swap balls. Then, Bob and Alice swap balls. Then, Alice and Bob swap balls. Then, Alice and Fred swap balls. Then, Fred and Eve swap balls. Then, Eve and Fred swap balls. Then, Fred and Alice swap balls. Then, Gertrude and Bob swap balls. Then, Bob and Fred swap balls. Then, Fred and Bob swap balls. Then, Bob and Gertrude swap balls. Then, Fred and Alice swap balls. Then, Alice and Eve swap balls. Then, Eve and Alice swap balls. Then, Alice and Fred swap balls. Then, nothing happens for a while. Then, Dave and Fred swap balls. Then, Dave and Fred discuss something. Then, Dave and Fred swap balls. Then, Bob and Alice swap balls. Then, Bob and Alice discuss something. Then, Bob and Alice swap balls. Then, nothing happens for a while. Then, Fred and Dave swap balls. Then, Dave and Bob swap balls. Then, Bob and Dave swap balls. Then, Dave and Fred swap balls. Then, Eve and Fred swap balls. Then, Fred and Eve swap balls. Then, Dave and Gertrude swap balls. Then, Dave and Gertrude discuss something. Then, Dave and Gertrude swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Eve swap balls. Then, Eve and Claire swap balls. Then, Claire and Eve swap balls. Then, Eve and Alice swap balls. Then, nothing happens for a while. Then, Eve and Fred swap balls. Then, Fred and Claire swap balls. Then, Claire and Fred swap balls. Then, Fred and Eve swap balls. Then, Eve and Dave swap balls. Then, Eve and Dave discuss something. Then, Eve and Dave swap balls. Then, nothing happens for a while. Then, Alice and Fred swap balls. Then, Fred and Alice swap balls. Then, nothing happens for a while. Then, Alice and Dave swap balls. Then, Dave and Gertrude swap balls. Then, Gertrude and Dave swap balls. Then, Dave and Alice swap balls. Then, nothing happens for a while. Then, Claire and Gertrude swap balls. Then, Gertrude and Bob swap balls. Then, Bob and Gertrude swap balls. Then, Gertrude and Claire swap balls. Then, Eve and Fred swap balls. Then, Fred and Eve swap balls. Then, Eve and Alice swap balls. Then, Eve and Alice discuss something. Then, Eve and Alice swap balls. Then, Bob and Gertrude swap balls. Then, Gertrude and Fred swap balls. Then, Fred and Gertrude swap balls. Then, Gertrude and Bob swap balls. Then, Bob and Gertrude swap balls. Then, Gertrude and Bob swap balls. Then, Eve and Fred swap balls. Then, Eve and Fred discuss something. Then, Eve and Fred swap balls. Then, Dave and Gertrude swap balls. Then, Gertrude and Bob swap balls. Then, Bob and Gertrude swap balls. Then, Gertrude and Dave swap balls. Then, Fred and Dave swap balls. Then, Fred and Dave discuss something. Then, Fred and Dave swap balls. Then, Gertrude and Claire swap balls. Then, Claire and Gertrude swap balls. Then, Bob and Claire swap balls. Then, Claire and Bob swap balls. Then, nothing happens for a while. Then, Dave and Alice swap balls. Then, Alice and Gertrude swap balls. Then, Gertrude and Alice swap balls. Then, Alice and Dave swap balls. Then, nothing happens for a while. Then, Alice and Dave swap balls. Then, Dave and Claire swap balls. Then, Claire and Dave swap balls. Then, Dave and Alice swap balls. Then, Eve and Alice swap balls. Then, Eve and Alice discuss something. Then, Eve and Alice swap balls. Then, Gertrude and Claire swap balls. Then, Alice and Claire swap balls. Then, Alice and Claire discuss something. Then, Alice and Claire swap balls. Then, nothing happens for a while. Then, Alice and Dave swap balls. Then, Dave and Alice swap balls. Then, Fred and Dave swap balls. Then, Dave and Fred swap balls. Then, Eve and Gertrude swap balls. Then, Gertrude and Dave swap balls. Then, Dave and Gertrude swap balls. Then, Gertrude and Eve swap balls. Then, Fred and Eve swap balls. Then, Fred and Eve discuss something. Then, Fred and Eve swap balls. Then, Eve and Gertrude swap balls. Then, Eve and Gertrude discuss something. Then, Eve and Gertrude swap balls. Then, Alice and Claire swap balls. Then, Claire and Alice swap balls. Then, Eve and Gertrude swap balls. Then, Gertrude and Claire swap balls. Then, Claire and Gertrude swap balls. Then, Gertrude and Eve swap balls. Then, Claire and Dave swap balls. Then, Dave and Claire swap balls. Then, Fred and Eve swap balls. Then, Eve and Claire swap balls. Then, Claire and Eve swap balls. Then, Eve and Fred swap balls. Then, Dave and Alice swap balls. Then, Alice and Dave swap balls. Then, Dave and Gertrude swap balls. Then, Gertrude and Dave swap balls. Then, Gertrude and Alice swap balls. Then, Alice and Bob swap balls. Then, Bob and Alice swap balls. Then, Alice and Gertrude swap balls. Then, Eve and Alice swap balls. Then, Eve and Alice discuss something. Then, Eve and Alice swap balls. Then, Bob and Alice swap balls. Then, Alice and Claire swap balls. Then, Claire and Alice swap balls. Then, Alice and Bob swap balls. Then, nothing happens for a while. Then, Gertrude and Alice swap balls. Then, Gertrude and Alice discuss something. Then, Gertrude and Alice swap balls. Then, Bob and Eve swap balls. Then, Eve and Alice swap balls. Then, Alice and Eve swap balls. Then, Eve and Bob swap balls. Then, nothing happens for a while. Then, Claire and Alice swap balls. Then, Alice and Fred swap balls. Then, Fred and Alice swap balls. Then, Alice and Claire swap balls. Then, Fred and Dave swap balls. Then, Dave and Fred swap balls. Then, Dave and Bob swap balls. Then, Alice and Gertrude swap balls. Then, Alice and Gertrude discuss something. Then, Alice and Gertrude swap balls. Then, Claire and Alice swap balls. Then, Alice and Claire swap balls. Then, Claire and Fred swap balls. Then, Fred and Bob swap balls. Then, Bob and Fred swap balls. Then, Fred and Claire swap balls. Then, Eve and Bob swap balls. Then, Bob and Fred swap balls. Then, Fred and Bob swap balls. Then, Bob and Eve swap balls. Then, Dave and Fred swap balls. Then, Fred and Alice swap balls. Then, Alice and Fred swap balls. Then, Fred and Dave swap balls. Then, Alice and Gertrude swap balls. Then, Alice and Gertrude discuss something. Then, Alice and Gertrude swap balls. Then, Alice and Claire swap balls. Then, Alice and Claire discuss something. Then, Alice and Claire swap balls. Then, nothing happens for a while. Then, Alice and Gertrude swap balls. Then, Gertrude and Bob swap balls. Then, Bob and Gertrude swap balls. Then, Gertrude and Alice swap balls. Then, Gertrude and Eve swap balls. Then, Eve and Bob swap balls. Then, Bob and Eve swap balls. Then, Eve and Gertrude swap balls. Then, Claire and Gertrude swap balls. Then, Gertrude and Eve swap balls. Then, Eve and Gertrude swap balls. Then, Gertrude and Claire swap balls. Then, nothing happens for a while. Then, Fred and Dave swap balls. Then, Dave and Fred swap balls. Then, Gertrude and Bob swap balls. Then, Bob and Claire swap balls. Then, Claire and Bob swap balls. Then, Bob and Gertrude swap balls. Then, Claire and Bob swap balls. Then, Claire and Bob discuss something. Then, Claire and Bob swap balls. Then, nothing happens for a while. Then, Fred and Claire swap balls. Then, Claire and Alice swap balls. Then, Alice and Claire swap balls. Then, Claire and Fred swap balls. Then, Gertrude and Eve swap balls. Then, Gertrude and Eve discuss something. Then, Gertrude and Eve swap balls. Then, Fred and Eve swap balls. Then, Eve and Fred swap balls. Then, Claire and Bob swap balls. Then, Bob and Claire swap balls. Then, Gertrude and Bob swap balls. Then, Bob and Gertrude swap balls. Then, Claire and Eve swap balls. Then, Eve and Claire swap balls. Then, Fred and Alice swap balls. Then, Alice and Bob swap balls. Then, Bob and Alice swap balls. Then, Alice and Fred swap balls. Then, Bob and Alice swap balls. Then, Alice and Fred swap balls. Then, Fred and Alice swap balls. Then, Alice and Bob swap balls. Then, Claire and Gertrude swap balls. Then, nothing happens for a while. Then, Gertrude and Eve swap balls. Then, Eve and Gertrude swap balls. Then, Gertrude and Alice swap balls. Then, Alice and Gertrude swap balls. Then, Alice and Claire swap balls. Then, Alice and Claire discuss something. Then, Alice and Claire swap balls. Then, Fred and Gertrude swap balls. Then, Fred and Gertrude discuss something. Then, Fred and Gertrude swap balls. Then, Eve and Dave swap balls. Then, Dave and Claire swap balls. Then, Claire and Dave swap balls. Then, Dave and Eve swap balls. Then, Gertrude and Fred swap balls. Then, Fred and Gertrude swap balls. Then, Gertrude and Alice swap balls. Then, Gertrude and Alice discuss something. Then, Gertrude and Alice swap balls. Then, Gertrude and Bob swap balls. Then, Bob and Claire swap balls. Then, Claire and Bob swap balls. Then, Bob and Gertrude swap balls. Then, nothing happens for a while. Then, Gertrude and Dave swap balls. Then, Dave and Gertrude swap balls. Then, nothing happens for a while. Then, Dave and Fred swap balls. Then, Dave and Fred discuss something. Then, Dave and Fred swap balls. Then, Claire and Eve swap balls. Then, Eve and Alice swap balls. Then, Alice and Eve swap balls. Then, Eve and Claire swap balls. Then, nothing happens for a while. Then, Claire and Eve swap balls. Then, Eve and Fred swap balls. Then, Fred and Eve swap balls. Then, Eve and Claire swap balls. Then, nothing happens for a while. Then, Dave and Claire swap balls. Then, Dave and Claire discuss something. Then, Dave and Claire swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, Eve and Gertrude swap balls. Then, Eve and Gertrude discuss something. Then, Eve and Gertrude swap balls. Then, Bob and Alice swap balls. Then, Alice and Eve swap balls. Then, Eve and Alice swap balls. Then, Alice and Bob swap balls. Then, Bob and Gertrude swap balls. Then, Gertrude and Claire swap balls. Then, Claire and Gertrude swap balls. Then, Gertrude and Bob swap balls. Then, Eve and Bob swap balls. Then, Bob and Alice swap balls. Then, Alice and Bob swap balls. Then, Bob and Eve swap balls. Then, Eve and Alice swap balls. Then, Eve and Alice discuss something. Then, Eve and Alice swap balls. Then, Gertrude and Fred swap balls. Then, Gertrude and Fred discuss something. Then, Gertrude and Fred swap balls. Then, Dave and Eve swap balls. Then, Eve and Fred swap balls. Then, Fred and Eve swap balls. Then, Eve and Dave swap balls. Then, Dave and Eve swap balls. Then, Dave and Eve discuss something. Then, Dave and Eve swap balls. Then, nothing happens for a while. Then, Gertrude and Fred swap balls. Then, Gertrude and Fred discuss something. Then, Gertrude and Fred swap balls. Then, Alice and Gertrude swap balls. Then, Gertrude and Claire swap balls. Then, Claire and Gertrude swap balls. Then, Gertrude and Alice swap balls. Then, Alice and Bob swap balls. Then, Bob and Alice swap balls. Then, Gertrude and Fred swap balls. Then, Gertrude and Fred discuss something. Then, Gertrude and Fred swap balls. Then, Dave and Fred swap balls. Then, Dave and Fred discuss something. Then, Dave and Fred swap balls. Then, nothing happens for a while. Then, Claire and Fred swap balls. Then, Claire and Fred discuss something. Then, Claire and Fred swap balls. Then, nothing happens for a while. Then, Gertrude and Alice swap balls. Then, Alice and Bob swap balls. Then, Bob and Alice swap balls. Then, Alice and Gertrude swap balls. Then, Alice and Dave swap balls. Then, Alice and Dave discuss something. Then, Alice and Dave swap balls. Then, Gertrude and Fred swap balls. Then, Fred and Claire swap balls. Then, Claire and Fred swap balls. Then, Fred and Gertrude swap balls. Then, Gertrude and Dave swap balls. Then, Dave and Claire swap balls. Then, Claire and Dave swap balls. Then, Dave and Gertrude swap balls. Then, Gertrude and Fred swap balls. Then, Fred and Dave swap balls. Then, Dave and Fred swap balls. Then, Fred and Gertrude swap balls. Then, Claire and Alice swap balls. Then, Alice and Claire swap balls. Then, Gertrude and Bob swap balls. Then, Bob and Gertrude swap balls. Then, Alice and Fred swap balls. Then, Fred and Alice swap balls. Then, Alice and Fred swap balls. Then, Bob and Alice swap balls. Finally, Eve and Bob swap balls. At the end of the game, Alice has the\\nOptions:\\n(A) white ball\\n(B) brown ball\\n(C) blue ball\\n(D) pink ball\\n(E) purple ball\\n(F) black ball\\n(G) yellow ball\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}