{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9e839265-0fc8-533f-802d-4a0e70601b2a","task_key":"bbeh~5fshuffled~5fobjects--full--9e839265-0fc8-533f-802d-4a0e70601b2a","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are holding a white elephant…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a yellow present, Bob has a orange ball, Claire has a green present, Dave has a pink ball, Eve has a brown present, Fred has a white present, and Gertrude has a purple present.\\nAs the event progresses, pairs of people swap gifts. First, Alice and Eve swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, Fred and Dave swap their gifts. Then, Dave and Alice swap their gifts. Then, Alice and Dave swap their gifts. Then, Dave and Fred swap their gifts. Then, Gertrude and Bob swap their gifts. Then, Bob and Claire swap their gifts. Then, Claire and Bob swap their gifts. Then, Bob and Gertrude swap their gifts. Then, Dave and Fred swap their gifts. Then, Fred and Bob swap their gifts. Then, Bob and Fred swap their gifts. Then, Fred and Dave swap their gifts. Then, Eve and Alice swap their gifts. Then, Alice and Dave swap their gifts. Then, Dave and Alice swap their gifts. Then, Alice and Eve swap their gifts. Then, Fred and Claire swap their gifts. Then, Claire and Gertrude swap their gifts. Then, Gertrude and Claire swap their gifts. Then, Claire and Fred swap their gifts. Then, Fred and Eve swap their gifts. Then, Eve and Dave swap their gifts. Then, Dave and Eve swap their gifts. Then, Eve and Fred swap their gifts. Then, nothing happens for a while. Then, Claire and Alice swap their gifts. Then, Claire and Alice discuss something. Then, Claire and Alice swap their gifts. Then, Claire and Gertrude swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Claire swap their gifts. Then, Gertrude and Eve swap their gifts. Then, Gertrude and Eve discuss something. Then, Gertrude and Eve swap their gifts. Then, Alice and Eve swap their gifts. Then, Eve and Dave swap their gifts. Then, Dave and Eve swap their gifts. Then, Eve and Alice swap their gifts. Then, nothing happens for a while. Then, Bob and Claire swap their gifts. Then, Bob and Claire discuss something. Then, Bob and Claire swap their gifts. Then, Fred and Claire swap their gifts. Then, Fred and Claire discuss something. Then, Fred and Claire swap their gifts. Then, Dave and Claire swap their gifts. Then, Claire and Fred swap their gifts. Then, Fred and Claire swap their gifts. Then, Claire and Dave swap their gifts. Then, Claire and Gertrude swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Claire swap their gifts. Then, Claire and Dave swap their gifts. Then, Claire and Dave discuss something. Then, Claire and Dave swap their gifts. Then, Bob and Gertrude swap their gifts. Then, Gertrude and Fred swap their gifts. Then, Fred and Gertrude swap their gifts. Then, Gertrude and Bob swap their gifts. Then, Eve and Fred swap their gifts. Then, Eve and Gertrude swap their gifts. Then, Gertrude and Claire swap their gifts. Then, Claire and Gertrude swap their gifts. Then, Gertrude and Eve swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Eve and Gertrude swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Eve swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Eve and Gertrude swap their gifts. Then, Eve and Gertrude discuss something. Then, Eve and Gertrude swap their gifts. Then, nothing happens for a while. Then, Claire and Eve swap their gifts. Then, Claire and Eve discuss something. Then, Claire and Eve swap their gifts. Then, Claire and Bob swap their gifts. Then, Claire and Bob discuss something. Then, Claire and Bob swap their gifts. Then, Bob and Dave swap their gifts. Then, Dave and Bob swap their gifts. Then, nothing happens for a while. Then, Bob and Eve swap their gifts. Then, Eve and Bob swap their gifts. Then, Gertrude and Bob swap their gifts. Then, Bob and Eve swap their gifts. Then, Eve and Bob swap their gifts. Then, Bob and Gertrude swap their gifts. Then, Bob and Alice swap their gifts. Then, Alice and Bob swap their gifts. Then, Bob and Eve swap their gifts. Then, Eve and Bob swap their gifts. Then, Dave and Eve swap their gifts. Then, Eve and Dave swap their gifts. Then, Bob and Eve swap their gifts. Then, Bob and Eve discuss something. Then, Bob and Eve swap their gifts. Then, Gertrude and Bob swap their gifts. Then, Gertrude and Bob discuss something. Then, Gertrude and Bob swap their gifts. Then, nothing happens for a while. Then, Eve and Alice swap their gifts. Then, Alice and Dave swap their gifts. Then, Dave and Alice swap their gifts. Then, Alice and Eve swap their gifts. Then, nothing happens for a while. Then, Gertrude and Claire swap their gifts. Then, Claire and Dave swap their gifts. Then, Dave and Claire swap their gifts. Then, Claire and Gertrude swap their gifts. Then, Bob and Gertrude swap their gifts. Then, Fred and Dave swap their gifts. Then, Dave and Gertrude swap their gifts. Then, Gertrude and Dave swap their gifts. Then, Dave and Fred swap their gifts. Then, Claire and Eve swap their gifts. Then, Claire and Eve discuss something. Then, Claire and Eve swap their gifts. Then, Fred and Alice swap their gifts. Then, Alice and Fred swap their gifts. Then, Dave and Alice swap their gifts. Then, Alice and Dave swap their gifts. Then, Bob and Dave swap their gifts. Then, Bob and Dave discuss something. Then, Bob and Dave swap their gifts. Then, Fred and Bob swap their gifts. Then, Bob and Fred swap their gifts. Then, Eve and Dave swap their gifts. Then, Eve and Dave discuss something. Then, Eve and Dave swap their gifts. Then, Eve and Alice swap their gifts. Then, Eve and Alice discuss something. Then, Eve and Alice swap their gifts. Then, Claire and Alice swap their gifts. Then, Alice and Fred swap their gifts. Then, Fred and Alice swap their gifts. Then, Alice and Claire swap their gifts. Then, Bob and Fred swap their gifts. Then, Fred and Bob swap their gifts. Then, Dave and Fred swap their gifts. Then, Dave and Fred discuss something. Then, Dave and Fred swap their gifts. Then, Dave and Gertrude swap their gifts. Then, Gertrude and Dave swap their gifts. Then, Fred and Gertrude swap their gifts. Then, Fred and Gertrude discuss something. Then, Fred and Gertrude swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Eve and Dave swap their gifts. Then, Dave and Eve swap their gifts. Then, Claire and Alice swap their gifts. Then, Claire and Alice discuss something. Then, Claire and Alice swap their gifts. Then, Dave and Bob swap their gifts. Then, Dave and Bob discuss something. Then, Dave and Bob swap their gifts. Then, Dave and Claire swap their gifts. Then, Claire and Eve swap their gifts. Then, Eve and Claire swap their gifts. Then, Claire and Dave swap their gifts. Then, Bob and Claire swap their gifts. Then, Claire and Gertrude swap their gifts. Then, Gertrude and Claire swap their gifts. Then, Claire and Bob swap their gifts. Then, Gertrude and Fred swap their gifts. Then, Fred and Alice swap their gifts. Then, Alice and Fred swap their gifts. Then, Fred and Gertrude swap their gifts. Then, Bob and Claire swap their gifts. Then, Bob and Claire discuss something. Then, Bob and Claire swap their gifts. Then, Eve and Bob swap their gifts. Then, Bob and Dave swap their gifts. Then, Dave and Bob swap their gifts. Then, Bob and Eve swap their gifts. Then, nothing happens for a while. Then, Fred and Gertrude swap their gifts. Then, Gertrude and Claire swap their gifts. Then, Claire and Gertrude swap their gifts. Then, Gertrude and Fred swap their gifts. Then, Bob and Claire swap their gifts. Then, Claire and Alice swap their gifts. Then, Alice and Claire swap their gifts. Then, Claire and Bob swap their gifts. Then, Claire and Dave swap their gifts. Then, Claire and Dave discuss something. Then, Claire and Dave swap their gifts. Then, Eve and Bob swap their gifts. Then, Bob and Eve swap their gifts. Then, Eve and Dave swap their gifts. Then, Dave and Fred swap their gifts. Then, Fred and Dave swap their gifts. Then, Dave and Eve swap their gifts. Then, Gertrude and Eve swap their gifts. Then, Eve and Gertrude swap their gifts. Then, Fred and Eve swap their gifts. Then, Dave and Gertrude swap their gifts. Then, Gertrude and Fred swap their gifts. Then, Fred and Gertrude swap their gifts. Then, Gertrude and Dave swap their gifts. Then, Eve and Alice swap their gifts. Then, Eve and Alice discuss something. Then, Eve and Alice swap their gifts. Then, Gertrude and Dave swap their gifts. Then, Gertrude and Dave discuss something. Then, Gertrude and Dave swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Alice and Claire swap their gifts. Then, Claire and Alice swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Fred and Bob swap their gifts. Then, Fred and Bob discuss something. Then, Fred and Bob swap their gifts. Then, Bob and Gertrude swap their gifts. Then, Gertrude and Bob swap their gifts. Then, Eve and Fred swap their gifts. Then, Eve and Fred discuss something. Then, Eve and Fred swap their gifts. Then, Eve and Alice swap their gifts. Then, Eve and Alice discuss something. Then, Eve and Alice swap their gifts. Then, Alice and Eve swap their gifts. Then, Eve and Fred swap their gifts. Then, Fred and Eve swap their gifts. Then, Eve and Alice swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Alice and Gertrude discuss something. Then, Alice and Gertrude swap their gifts. Then, nothing happens for a while. Then, Claire and Alice swap their gifts. Then, Alice and Claire swap their gifts. Then, nothing happens for a while. Then, Dave and Claire swap their gifts. Then, Claire and Fred swap their gifts. Then, Fred and Claire swap their gifts. Then, Claire and Dave swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, Eve and Alice swap their gifts. Then, Alice and Claire swap their gifts. Then, Claire and Alice swap their gifts. Then, Alice and Eve swap their gifts. Then, Alice and Claire swap their gifts. Then, Claire and Gertrude swap their gifts. Then, Gertrude and Claire swap their gifts. Then, Claire and Alice swap their gifts. Then, Alice and Fred swap their gifts. Then, Alice and Fred discuss something. Then, Alice and Fred swap their gifts. Then, Eve and Claire swap their gifts. Then, Claire and Eve swap their gifts. Then, Alice and Eve swap their gifts. Then, Alice and Eve discuss something. Then, Alice and Eve swap their gifts. Then, Bob and Alice swap their gifts. Then, Bob and Alice discuss something. Then, Bob and Alice swap their gifts. Then, Eve and Dave swap their gifts. Then, Dave and Eve swap their gifts. Then, Alice and Fred swap their gifts. Then, Fred and Alice swap their gifts. Then, nothing happens for a while. Then, Dave and Claire swap their gifts. Then, Claire and Gertrude swap their gifts. Then, Gertrude and Claire swap their gifts. Then, Claire and Dave swap their gifts. Then, nothing happens for a while. Then, Dave and Bob swap their gifts. Then, Bob and Fred swap their gifts. Then, Fred and Bob swap their gifts. Then, Bob and Dave swap their gifts. Then, Claire and Eve swap their gifts. Then, Claire and Eve discuss something. Then, Claire and Eve swap their gifts. Then, nothing happens for a while. Then, Eve and Claire swap their gifts. Then, Claire and Eve swap their gifts. Then, Dave and Alice swap their gifts. Then, Claire and Eve swap their gifts. Finally, Gertrude and Eve swap their gifts. At the end of the event, Gertrude has the\\nOptions:\\n(A) yellow present\\n(B) orange ball\\n(C) green present\\n(D) pink ball\\n(E) brown present\\n(F) white present\\n(G) purple present\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}