{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"75a06068-c437-507a-9219-376efa606ea4","task_key":"bbeh~5fshuffled~5fobjects--full--75a06068-c437-507a-9219-376efa606ea4","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are playing a game. At the…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are playing a game. At the start of the game, they are each holding a ball: Alice has a orange ball, Bob has a yellow ball, Claire has a brown ball, Dave has a white ball, Eve has a black ball, Fred has a red ball, and Gertrude has a purple ball.\\nAs the game progresses, pairs of players trade balls. First, Claire and Bob swap balls. Then, Gertrude and Dave swap balls. Then, Dave and Gertrude swap balls. Then, Dave and Claire swap balls. Then, Claire and Alice swap balls. Then, Alice and Claire swap balls. Then, Claire and Dave swap balls. Then, Bob and Claire swap balls. Then, Claire and Alice swap balls. Then, Alice and Claire swap balls. Then, Claire and Bob swap balls. Then, Eve and Claire swap balls. Then, Eve and Claire discuss something. Then, Eve and Claire swap balls. Then, Dave and Gertrude swap balls. Then, Dave and Gertrude discuss something. Then, Dave and Gertrude swap balls. Then, nothing happens for a while. Then, Gertrude and Claire swap balls. Then, Claire and Eve swap balls. Then, Eve and Claire swap balls. Then, Claire and Gertrude swap balls. Then, nothing happens for a while. Then, Eve and Alice swap balls. Then, Alice and Fred swap balls. Then, Fred and Alice swap balls. Then, Alice and Eve swap balls. Then, Claire and Fred swap balls. Then, Fred and Claire swap balls. Then, Eve and Dave swap balls. Then, Eve and Dave discuss something. Then, Eve and Dave swap balls. Then, Dave and Gertrude swap balls. Then, Gertrude and Dave swap balls. Then, Alice and Gertrude swap balls. Then, Gertrude and Eve swap balls. Then, Eve and Gertrude swap balls. Then, Gertrude and Alice swap balls. Then, Eve and Alice swap balls. Then, Alice and Eve swap balls. Then, Fred and Claire swap balls. Then, Fred and Claire discuss something. Then, Fred and Claire swap balls. Then, nothing happens for a while. Then, Fred and Claire swap balls. Then, Fred and Claire discuss something. Then, Fred and Claire swap balls. Then, nothing happens for a while. Then, Dave and Alice swap balls. Then, Dave and Alice discuss something. Then, Dave and Alice swap balls. Then, Dave and Fred swap balls. Then, Fred and Eve swap balls. Then, Eve and Fred swap balls. Then, Fred and Dave swap balls. Then, Claire and Gertrude swap balls. Then, Claire and Gertrude discuss something. Then, Claire and Gertrude swap balls. Then, Eve and Fred swap balls. Then, Fred and Eve swap balls. Then, nothing happens for a while. Then, Claire and Dave swap balls. Then, Claire and Dave swap balls. Then, Claire and Dave discuss something. Then, Claire and Dave swap balls. Then, Alice and Dave swap balls. Then, Alice and Dave discuss something. Then, Alice and Dave swap balls. Then, Gertrude and Claire swap balls. Then, Claire and Gertrude swap balls. Then, nothing happens for a while. Then, Gertrude and Bob swap balls. Then, Bob and Eve swap balls. Then, Eve and Bob swap balls. Then, Bob and Gertrude swap balls. Then, Eve and Dave swap balls. Then, Dave and Eve swap balls. Then, Eve and Fred swap balls. Then, Fred and Eve swap balls. Then, Fred and Dave swap balls. Then, Dave and Alice swap balls. Then, Alice and Dave swap balls. Then, Dave and Fred swap balls. Then, Claire and Alice swap balls. Then, Claire and Alice discuss something. Then, Claire and Alice swap balls. Then, Eve and Gertrude swap balls. Then, Eve and Gertrude discuss something. Then, Eve and Gertrude swap balls. Then, Bob and Fred swap balls. Then, Fred and Bob swap balls. Then, nothing happens for a while. Then, Dave and Alice swap balls. Then, Alice and Eve swap balls. Then, Eve and Alice swap balls. Then, Alice and Dave swap balls. Then, Eve and Gertrude swap balls. Then, Gertrude and Eve swap balls. Then, Claire and Gertrude swap balls. Then, Claire and Gertrude discuss something. Then, Claire and Gertrude swap balls. Then, Fred and Alice swap balls. Then, Alice and Claire swap balls. Then, Claire and Alice swap balls. Then, Alice and Fred swap balls. Then, Dave and Gertrude swap balls. Then, Gertrude and Alice swap balls. Then, Alice and Gertrude swap balls. Then, Gertrude and Dave swap balls. Then, Dave and Gertrude swap balls. Then, Dave and Gertrude discuss something. Then, Dave and Gertrude swap balls. Then, nothing happens for a while. Then, Claire and Bob swap balls. Then, Bob and Alice swap balls. Then, Alice and Bob swap balls. Then, Bob and Claire swap balls. Then, nothing happens for a while. Then, Alice and Bob swap balls. Then, Bob and Alice swap balls. Then, Fred and Gertrude swap balls. Then, Alice and Dave swap balls. Then, Dave and Fred swap balls. Then, Fred and Dave swap balls. Then, Dave and Alice swap balls. Then, nothing happens for a while. Then, Alice and Bob swap balls. Then, Bob and Fred swap balls. Then, Fred and Bob swap balls. Then, Bob and Alice swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, Gertrude and Dave swap balls. Then, Gertrude and Dave discuss something. Then, Gertrude and Dave swap balls. Then, Fred and Bob swap balls. Then, Fred and Bob discuss something. Then, Fred and Bob swap balls. Then, Gertrude and Alice swap balls. Then, Alice and Gertrude swap balls. Then, Bob and Fred swap balls. Then, Bob and Fred discuss something. Then, Bob and Fred swap balls. Then, Dave and Gertrude swap balls. Then, Gertrude and Bob swap balls. Then, Bob and Gertrude swap balls. Then, Gertrude and Dave swap balls. Then, Bob and Eve swap balls. Then, Eve and Dave swap balls. Then, Dave and Eve swap balls. Then, Eve and Bob swap balls. Then, Bob and Alice swap balls. Then, Alice and Claire swap balls. Then, Claire and Alice swap balls. Then, Alice and Bob swap balls. Then, Bob and Claire swap balls. Then, Bob and Claire discuss something. Then, Bob and Claire swap balls. Then, Dave and Bob swap balls. Then, Bob and Dave swap balls. Then, Dave and Eve swap balls. Then, Eve and Alice swap balls. Then, Alice and Eve swap balls. Then, Eve and Dave swap balls. Then, Gertrude and Claire swap balls. Then, Claire and Gertrude swap balls. Then, Bob and Dave swap balls. Then, Dave and Fred swap balls. Then, Fred and Dave swap balls. Then, Dave and Bob swap balls. Then, Alice and Claire swap balls. Then, Claire and Gertrude swap balls. Then, Gertrude and Claire swap balls. Then, Claire and Alice swap balls. Then, Claire and Fred swap balls. Then, Fred and Claire swap balls. Then, Bob and Dave swap balls. Then, Dave and Bob swap balls. Then, Eve and Fred swap balls. Then, Eve and Fred discuss something. Then, Eve and Fred swap balls. Then, Gertrude and Bob swap balls. Then, Gertrude and Bob discuss something. Then, Gertrude and Bob swap balls. Then, Dave and Fred swap balls. Then, Fred and Gertrude swap balls. Then, Gertrude and Fred swap balls. Then, Fred and Dave swap balls. Then, Eve and Alice swap balls. Then, Alice and Dave swap balls. Then, Dave and Alice swap balls. Then, Alice and Eve swap balls. Then, Fred and Alice swap balls. Then, Fred and Alice discuss something. Then, Fred and Alice swap balls. Then, Claire and Eve swap balls. Then, Eve and Claire swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Claire and Fred swap balls. Then, Fred and Claire swap balls. Then, Fred and Claire swap balls. Then, Fred and Claire discuss something. Then, Fred and Claire swap balls. Then, Alice and Eve swap balls. Then, Eve and Alice swap balls. Then, Eve and Alice swap balls. Then, Alice and Eve swap balls. Then, Dave and Claire swap balls. Then, Claire and Dave swap balls. Then, Eve and Fred swap balls. Then, Eve and Fred discuss something. Then, Eve and Fred swap balls. Then, Eve and Claire swap balls. Then, nothing happens for a while. Then, Claire and Dave swap balls. Then, Claire and Dave discuss something. Then, Claire and Dave swap balls. Then, Claire and Eve swap balls. Then, Eve and Gertrude swap balls. Then, Gertrude and Eve swap balls. Then, Eve and Claire swap balls. Then, nothing happens for a while. Then, Claire and Fred swap balls. Then, Fred and Claire swap balls. Then, Gertrude and Alice swap balls. Then, Alice and Gertrude swap balls. Then, Bob and Claire swap balls. Then, Bob and Claire discuss something. Then, Bob and Claire swap balls. Then, Fred and Gertrude swap balls. Then, Fred and Gertrude discuss something. Then, Fred and Gertrude swap balls. Then, Claire and Dave swap balls. Then, Claire and Dave discuss something. Then, Claire and Dave swap balls. Then, Claire and Alice swap balls. Then, Alice and Bob swap balls. Then, Bob and Alice swap balls. Then, Alice and Claire swap balls. Then, Alice and Bob swap balls. Then, Bob and Alice swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Gertrude swap balls. Then, Gertrude and Eve swap balls. Then, Eve and Gertrude swap balls. Then, Gertrude and Dave swap balls. Then, nothing happens for a while. Then, Dave and Eve swap balls. Then, Eve and Alice swap balls. Then, Alice and Eve swap balls. Then, Eve and Dave swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Claire swap balls. Then, Claire and Alice swap balls. Then, Bob and Alice swap balls. Then, Alice and Gertrude swap balls. Then, Gertrude and Alice swap balls. Then, Alice and Bob swap balls. Then, nothing happens for a while. Then, Eve and Alice swap balls. Then, Alice and Fred swap balls. Then, Fred and Alice swap balls. Then, Alice and Eve swap balls. Then, Bob and Alice swap balls. Then, Alice and Bob swap balls. Then, Bob and Eve swap balls. Then, Eve and Bob swap balls. Then, Claire and Alice swap balls. Then, Claire and Alice discuss something. Then, Claire and Alice swap balls. Then, nothing happens for a while. Then, Eve and Dave swap balls. Then, Eve and Dave discuss something. Then, Eve and Dave swap balls. Then, Fred and Eve swap balls. Then, Eve and Fred swap balls. Then, Eve and Dave swap balls. Then, Dave and Alice swap balls. Then, Alice and Dave swap balls. Then, Dave and Eve swap balls. Then, Claire and Bob swap balls. Then, Claire and Bob discuss something. Then, Claire and Bob swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Fred and Alice swap balls. Then, Alice and Fred swap balls. Then, Bob and Eve swap balls. Then, Eve and Bob swap balls. Then, nothing happens for a while. Then, Gertrude and Fred swap balls. Then, Fred and Claire swap balls. Then, Claire and Fred swap balls. Then, Fred and Gertrude swap balls. Then, Fred and Eve swap balls. Then, Eve and Gertrude swap balls. Then, Gertrude and Eve swap balls. Then, Eve and Fred swap balls. Then, Eve and Fred swap balls. Then, Alice and Dave swap balls. Finally, Eve and Claire swap balls. At the end of the game, Eve has the\\nOptions:\\n(A) orange ball\\n(B) yellow ball\\n(C) brown ball\\n(D) white ball\\n(E) black ball\\n(F) red ball\\n(G) purple ball\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}