{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7e4d85de-f258-5d11-b357-87b97aef1b9a","task_key":"bbeh~5fshuffled~5fobjects--full--7e4d85de-f258-5d11-b357-87b97aef1b9a","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are playing a game. At the…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are playing a game. At the start of the game, they are each holding a ball: Alice has a orange ball, Bob has a black ball, Claire has a blue ball, Dave has a brown ball, Eve has a red ball, Fred has a purple ball, and Gertrude has a pink ball.\\nAs the game progresses, pairs of players trade balls. First, Gertrude and Eve swap balls. Then, Gertrude and Alice swap balls. Then, Alice and Gertrude swap balls. Then, Dave and Bob swap balls. Then, Bob and Fred swap balls. Then, Fred and Bob swap balls. Then, Bob and Dave swap balls. Then, Dave and Gertrude swap balls. Then, Dave and Gertrude discuss something. Then, Dave and Gertrude swap balls. Then, Eve and Fred swap balls. Then, Fred and Dave swap balls. Then, Dave and Fred swap balls. Then, Fred and Eve swap balls. Then, nothing happens for a while. Then, Dave and Fred swap balls. Then, Dave and Fred discuss something. Then, Dave and Fred swap balls. Then, Alice and Claire swap balls. Then, Alice and Claire discuss something. Then, Alice and Claire swap balls. Then, Bob and Eve swap balls. Then, Bob and Eve discuss something. Then, Bob and Eve swap balls. Then, Bob and Eve swap balls. Then, Eve and Bob swap balls. Then, Dave and Claire swap balls. Then, Claire and Dave swap balls. Then, Claire and Alice swap balls. Then, Claire and Alice discuss something. Then, Claire and Alice swap balls. Then, Bob and Fred swap balls. Then, Fred and Bob swap balls. Then, Claire and Bob swap balls. Then, Claire and Bob discuss something. Then, Claire and Bob swap balls. Then, Claire and Alice swap balls. Then, Claire and Alice discuss something. Then, Claire and Alice swap balls. Then, nothing happens for a while. Then, Eve and Dave swap balls. Then, Dave and Bob swap balls. Then, Bob and Dave swap balls. Then, Dave and Eve swap balls. Then, nothing happens for a while. Then, Fred and Alice swap balls. Then, Alice and Fred swap balls. Then, Gertrude and Alice swap balls. Then, Alice and Eve swap balls. Then, Eve and Alice swap balls. Then, Alice and Gertrude swap balls. Then, Claire and Bob swap balls. Then, Claire and Bob discuss something. Then, Claire and Bob swap balls. Then, Fred and Dave swap balls. Then, Dave and Fred swap balls. Then, Bob and Alice swap balls. Then, Alice and Eve swap balls. Then, Eve and Alice swap balls. Then, Alice and Bob swap balls. Then, Gertrude and Eve swap balls. Then, Eve and Claire swap balls. Then, Claire and Eve swap balls. Then, Eve and Gertrude swap balls. Then, Fred and Dave swap balls. Then, Dave and Gertrude swap balls. Then, Gertrude and Dave swap balls. Then, Dave and Fred swap balls. Then, Gertrude and Bob swap balls. Then, Bob and Gertrude swap balls. Then, Gertrude and Bob swap balls. Then, Gertrude and Bob discuss something. Then, Gertrude and Bob swap balls. Then, nothing happens for a while. Then, Claire and Eve swap balls. Then, Eve and Gertrude swap balls. Then, Gertrude and Eve swap balls. Then, Eve and Claire swap balls. Then, Fred and Bob swap balls. Then, Gertrude and Eve swap balls. Then, Eve and Dave swap balls. Then, Dave and Eve swap balls. Then, Eve and Gertrude swap balls. Then, Claire and Dave swap balls. Then, Claire and Dave discuss something. Then, Claire and Dave swap balls. Then, nothing happens for a while. Then, Claire and Gertrude swap balls. Then, Claire and Gertrude discuss something. Then, Claire and Gertrude swap balls. Then, nothing happens for a while. Then, Gertrude and Claire swap balls. Then, Claire and Gertrude swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Dave swap balls. Then, Dave and Fred swap balls. Then, Fred and Dave swap balls. Then, Dave and Alice swap balls. Then, Gertrude and Eve swap balls. Then, Eve and Bob swap balls. Then, Bob and Eve swap balls. Then, Eve and Gertrude swap balls. Then, Alice and Dave swap balls. Then, Dave and Alice swap balls. Then, Alice and Eve swap balls. Then, Eve and Dave swap balls. Then, Dave and Eve swap balls. Then, Eve and Alice swap balls. Then, Dave and Eve swap balls. Then, Eve and Claire swap balls. Then, Claire and Eve swap balls. Then, Eve and Dave swap balls. Then, Gertrude and Alice swap balls. Then, Alice and Bob swap balls. Then, Bob and Alice swap balls. Then, Alice and Gertrude swap balls. Then, Dave and Alice swap balls. Then, Alice and Dave swap balls. Then, nothing happens for a while. Then, Fred and Dave swap balls. Then, Dave and Eve swap balls. Then, Eve and Dave swap balls. Then, Dave and Fred swap balls. Then, Bob and Fred swap balls. Then, Bob and Fred discuss something. Then, Bob and Fred swap balls. Then, Gertrude and Alice swap balls. Then, Gertrude and Alice discuss something. Then, Gertrude and Alice swap balls. Then, Fred and Eve swap balls. Then, nothing happens for a while. Then, Alice and Dave swap balls. Then, Alice and Dave discuss something. Then, Alice and Dave swap balls. Then, Gertrude and Claire swap balls. Then, Claire and Gertrude swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Gertrude swap balls. Then, Alice and Gertrude discuss something. Then, Alice and Gertrude swap balls. Then, Eve and Alice swap balls. Then, Alice and Eve swap balls. Then, Dave and Eve swap balls. Then, Eve and Gertrude swap balls. Then, Gertrude and Eve swap balls. Then, Eve and Dave swap balls. Then, Alice and Eve swap balls. Then, Eve and Alice swap balls. Then, Dave and Gertrude swap balls. Then, Gertrude and Eve swap balls. Then, Eve and Gertrude swap balls. Then, Gertrude and Dave swap balls. Then, Gertrude and Dave swap balls. Then, Dave and Gertrude swap balls. Then, Gertrude and Eve swap balls. Then, Gertrude and Eve discuss something. Then, Gertrude and Eve swap balls. Then, Eve and Alice swap balls. Then, Eve and Alice discuss something. Then, Eve and Alice swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Alice swap balls. Then, Alice and Bob swap balls. Then, Bob and Alice swap balls. Then, Alice and Dave swap balls. Then, Eve and Bob swap balls. Then, Bob and Dave swap balls. Then, Dave and Bob swap balls. Then, Bob and Eve swap balls. Then, Gertrude and Alice swap balls. Then, Alice and Eve swap balls. Then, Eve and Alice swap balls. Then, Alice and Gertrude swap balls. Then, nothing happens for a while. Then, Dave and Alice swap balls. Then, Alice and Dave swap balls. Then, nothing happens for a while. Then, Eve and Bob swap balls. Then, Bob and Eve swap balls. Then, Alice and Dave swap balls. Then, Dave and Alice swap balls. Then, Gertrude and Claire swap balls. Then, Claire and Fred swap balls. Then, Fred and Claire swap balls. Then, Claire and Gertrude swap balls. Then, Eve and Gertrude swap balls. Then, Gertrude and Eve swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, Claire and Dave swap balls. Then, Dave and Bob swap balls. Then, Bob and Dave swap balls. Then, Dave and Claire swap balls. Then, Bob and Claire swap balls. Then, Claire and Bob swap balls. Then, nothing happens for a while. Then, Dave and Eve swap balls. Then, Eve and Fred swap balls. Then, Fred and Eve swap balls. Then, Eve and Dave swap balls. Then, Fred and Dave swap balls. Then, Dave and Fred swap balls. Then, nothing happens for a while. Then, Claire and Dave swap balls. Then, Dave and Bob swap balls. Then, Bob and Dave swap balls. Then, Dave and Claire swap balls. Then, Gertrude and Fred swap balls. Then, Fred and Gertrude swap balls. Then, Eve and Dave swap balls. Then, Dave and Eve swap balls. Then, nothing happens for a while. Then, Bob and Fred swap balls. Then, Fred and Bob swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Dave swap balls. Then, Alice and Dave discuss something. Then, Alice and Dave swap balls. Then, Dave and Claire swap balls. Then, Claire and Alice swap balls. Then, Alice and Claire swap balls. Then, Claire and Dave swap balls. Then, nothing happens for a while. Then, nothing happens for a while. Then, Gertrude and Eve swap balls. Then, Eve and Gertrude swap balls. Then, Claire and Alice swap balls. Then, Alice and Fred swap balls. Then, Fred and Alice swap balls. Then, Alice and Claire swap balls. Then, Bob and Dave swap balls. Then, Bob and Dave discuss something. Then, Bob and Dave swap balls. Then, Bob and Gertrude swap balls. Then, Gertrude and Bob swap balls. Then, Alice and Gertrude swap balls. Then, Fred and Alice swap balls. Then, Alice and Fred swap balls. Then, Fred and Claire swap balls. Then, Claire and Dave swap balls. Then, Dave and Claire swap balls. Then, Claire and Fred swap balls. Then, Dave and Eve swap balls. Then, Eve and Fred swap balls. Then, Fred and Eve swap balls. Then, Eve and Dave swap balls. Then, Dave and Gertrude swap balls. Then, Gertrude and Eve swap balls. Then, Eve and Gertrude swap balls. Then, Gertrude and Dave swap balls. Then, nothing happens for a while. Then, Gertrude and Alice swap balls. Then, Alice and Gertrude swap balls. Then, Dave and Claire swap balls. Then, Dave and Claire discuss something. Then, Dave and Claire swap balls. Then, Fred and Claire swap balls. Then, Fred and Claire discuss something. Then, Fred and Claire swap balls. Then, Gertrude and Eve swap balls. Then, Gertrude and Eve discuss something. Then, Gertrude and Eve swap balls. Then, nothing happens for a while. Then, Dave and Eve swap balls. Then, Dave and Eve discuss something. Then, Dave and Eve swap balls. Then, nothing happens for a while. Then, Dave and Bob swap balls. Then, Bob and Dave swap balls. Then, nothing happens for a while. Then, Bob and Gertrude swap balls. Then, Gertrude and Claire swap balls. Then, Claire and Gertrude swap balls. Then, Gertrude and Bob swap balls. Then, Bob and Eve swap balls. Then, Eve and Dave swap balls. Then, Dave and Eve swap balls. Then, Eve and Bob swap balls. Then, Dave and Bob swap balls. Then, Bob and Claire swap balls. Then, Claire and Bob swap balls. Then, Bob and Dave swap balls. Then, Eve and Fred swap balls. Then, Eve and Fred discuss something. Then, Eve and Fred swap balls. Then, Eve and Alice swap balls. Then, Alice and Gertrude swap balls. Then, Gertrude and Alice swap balls. Then, Alice and Eve swap balls. Then, nothing happens for a while. Then, Fred and Eve swap balls. Then, Fred and Eve discuss something. Then, Fred and Eve swap balls. Then, Gertrude and Eve swap balls. Then, Eve and Claire swap balls. Then, Claire and Eve swap balls. Then, Eve and Gertrude swap balls. Then, Gertrude and Alice swap balls. Then, Alice and Dave swap balls. Then, Dave and Alice swap balls. Then, Alice and Gertrude swap balls. Then, Dave and Gertrude swap balls. Then, Dave and Gertrude discuss something. Then, Dave and Gertrude swap balls. Then, Bob and Gertrude swap balls. Then, Gertrude and Bob swap balls. Then, Fred and Dave swap balls. Then, Dave and Fred swap balls. Then, Claire and Bob swap balls. Then, Bob and Dave swap balls. Then, Dave and Bob swap balls. Then, Bob and Claire swap balls. Then, Dave and Fred swap balls. Then, Fred and Alice swap balls. Then, Alice and Fred swap balls. Then, Fred and Dave swap balls. Then, Claire and Alice swap balls. Then, Alice and Gertrude swap balls. Then, Gertrude and Alice swap balls. Then, Alice and Claire swap balls. Then, Alice and Claire swap balls. Then, Claire and Eve swap balls. Then, Eve and Claire swap balls. Then, Claire and Alice swap balls. Then, Fred and Eve swap balls. Then, Eve and Dave swap balls. Then, Dave and Eve swap balls. Then, Eve and Fred swap balls. Then, Gertrude and Claire swap balls. Then, Gertrude and Dave swap balls. Finally, Alice and Bob swap balls. At the end of the game, Fred has the\\nOptions:\\n(A) orange ball\\n(B) black ball\\n(C) blue ball\\n(D) brown ball\\n(E) red ball\\n(F) purple ball\\n(G) pink ball\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}