{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"036050a6-8cbf-57ed-a21c-84018f3846f0","task_key":"bbeh~5fshuffled~5fobjects--full--036050a6-8cbf-57ed-a21c-84018f3846f0","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are holding a white elephant…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a blue present, Bob has a green present, Claire has a brown present, Dave has a purple present, Eve has a orange ball, Fred has a pink ball, and Gertrude has a red present.\\nAs the event progresses, pairs of people swap gifts. First, Eve and Claire swap their gifts. Then, nothing happens for a while. Then, Bob and Gertrude swap their gifts. Then, Gertrude and Claire swap their gifts. Then, Claire and Gertrude swap their gifts. Then, Gertrude and Bob swap their gifts. Then, Dave and Alice swap their gifts. Then, Alice and Claire swap their gifts. Then, Claire and Alice swap their gifts. Then, Alice and Dave swap their gifts. Then, Fred and Gertrude swap their gifts. Then, Gertrude and Bob swap their gifts. Then, Bob and Gertrude swap their gifts. Then, Gertrude and Fred swap their gifts. Then, Gertrude and Eve swap their gifts. Then, Eve and Gertrude swap their gifts. Then, Dave and Alice swap their gifts. Then, Alice and Bob swap their gifts. Then, Bob and Alice swap their gifts. Then, Alice and Dave swap their gifts. Then, Claire and Bob swap their gifts. Then, Claire and Bob discuss something. Then, Claire and Bob swap their gifts. Then, nothing happens for a while. Then, Gertrude and Eve swap their gifts. Then, Eve and Gertrude swap their gifts. Then, Fred and Dave swap their gifts. Then, Fred and Dave discuss something. Then, Fred and Dave swap their gifts. Then, Claire and Fred swap their gifts. Then, Fred and Claire swap their gifts. Then, Eve and Claire swap their gifts. Then, Claire and Dave swap their gifts. Then, Dave and Claire swap their gifts. Then, Claire and Eve swap their gifts. Then, Bob and Fred swap their gifts. Then, Fred and Claire swap their gifts. Then, Claire and Fred swap their gifts. Then, Fred and Bob swap their gifts. Then, Eve and Alice swap their gifts. Then, Alice and Eve swap their gifts. Then, Eve and Fred swap their gifts. Then, Fred and Eve swap their gifts. Then, Fred and Alice swap their gifts. Then, Alice and Bob swap their gifts. Then, Bob and Alice swap their gifts. Then, Alice and Fred swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, Bob and Fred swap their gifts. Then, Bob and Fred discuss something. Then, Bob and Fred swap their gifts. Then, Fred and Claire swap their gifts. Then, Claire and Gertrude swap their gifts. Then, Gertrude and Claire swap their gifts. Then, Claire and Fred swap their gifts. Then, Gertrude and Eve swap their gifts. Then, Eve and Gertrude swap their gifts. Then, Claire and Fred swap their gifts. Then, Claire and Fred discuss something. Then, Claire and Fred swap their gifts. Then, Gertrude and Dave swap their gifts. Then, Dave and Alice swap their gifts. Then, Alice and Dave swap their gifts. Then, Dave and Gertrude swap their gifts. Then, Alice and Dave swap their gifts. Then, Alice and Dave discuss something. Then, Alice and Dave swap their gifts. Then, nothing happens for a while. Then, Bob and Claire swap their gifts. Then, Claire and Bob swap their gifts. Then, Alice and Eve swap their gifts. Then, Eve and Fred swap their gifts. Then, Fred and Eve swap their gifts. Then, Eve and Alice swap their gifts. Then, Fred and Claire swap their gifts. Then, nothing happens for a while. Then, Bob and Alice swap their gifts. Then, Alice and Bob swap their gifts. Then, Bob and Gertrude swap their gifts. Then, Gertrude and Bob swap their gifts. Then, Dave and Gertrude swap their gifts. Then, Dave and Gertrude discuss something. Then, Dave and Gertrude swap their gifts. Then, Fred and Claire swap their gifts. Then, Fred and Claire discuss something. Then, Fred and Claire swap their gifts. Then, Gertrude and Dave swap their gifts. Then, Dave and Eve swap their gifts. Then, Eve and Dave swap their gifts. Then, Dave and Gertrude swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, Fred and Alice swap their gifts. Then, Fred and Alice discuss something. Then, Fred and Alice swap their gifts. Then, Gertrude and Dave swap their gifts. Then, Gertrude and Dave discuss something. Then, Gertrude and Dave swap their gifts. Then, Dave and Claire swap their gifts. Then, Claire and Dave swap their gifts. Then, Fred and Dave swap their gifts. Then, Dave and Alice swap their gifts. Then, Alice and Dave swap their gifts. Then, Dave and Fred swap their gifts. Then, Alice and Eve swap their gifts. Then, Eve and Bob swap their gifts. Then, Bob and Eve swap their gifts. Then, Eve and Alice swap their gifts. Then, Claire and Alice swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Alice and Claire swap their gifts. Then, nothing happens for a while. Then, Fred and Claire swap their gifts. Then, Claire and Bob swap their gifts. Then, Bob and Claire swap their gifts. Then, Claire and Fred swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Eve and Fred swap their gifts. Then, Fred and Claire swap their gifts. Then, Claire and Fred swap their gifts. Then, Fred and Eve swap their gifts. Then, Fred and Gertrude swap their gifts. Then, Gertrude and Dave swap their gifts. Then, Dave and Gertrude swap their gifts. Then, Gertrude and Fred swap their gifts. Then, Dave and Fred swap their gifts. Then, Fred and Dave swap their gifts. Then, Alice and Eve swap their gifts. Then, Eve and Claire swap their gifts. Then, Claire and Eve swap their gifts. Then, Eve and Alice swap their gifts. Then, nothing happens for a while. Then, Eve and Gertrude swap their gifts. Then, Gertrude and Eve swap their gifts. Then, Claire and Fred swap their gifts. Then, Fred and Claire swap their gifts. Then, Dave and Claire swap their gifts. Then, Claire and Dave swap their gifts. Then, Alice and Dave swap their gifts. Then, Alice and Dave discuss something. Then, Alice and Dave swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Eve swap their gifts. Then, Alice and Eve discuss something. Then, Alice and Eve swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, Gertrude and Bob swap their gifts. Then, Bob and Gertrude swap their gifts. Then, Bob and Claire swap their gifts. Then, Claire and Bob swap their gifts. Then, Alice and Claire swap their gifts. Then, Alice and Claire discuss something. Then, Alice and Claire swap their gifts. Then, Dave and Bob swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Claire swap their gifts. Then, Claire and Fred swap their gifts. Then, Fred and Claire swap their gifts. Then, Claire and Dave swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, Bob and Claire swap their gifts. Then, Claire and Eve swap their gifts. Then, Eve and Claire swap their gifts. Then, Claire and Bob swap their gifts. Then, Eve and Claire swap their gifts. Then, Eve and Claire discuss something. Then, Eve and Claire swap their gifts. Then, nothing happens for a while. Then, Fred and Alice swap their gifts. Then, Alice and Fred swap their gifts. Then, Gertrude and Eve swap their gifts. Then, Gertrude and Eve discuss something. Then, Gertrude and Eve swap their gifts. Then, nothing happens for a while. Then, Dave and Fred swap their gifts. Then, Fred and Claire swap their gifts. Then, Claire and Fred swap their gifts. Then, Fred and Dave swap their gifts. Then, Bob and Dave swap their gifts. Then, Bob and Dave discuss something. Then, Bob and Dave swap their gifts. Then, Eve and Alice swap their gifts. Then, Eve and Alice discuss something. Then, Eve and Alice swap their gifts. Then, Bob and Gertrude swap their gifts. Then, Gertrude and Fred swap their gifts. Then, Fred and Gertrude swap their gifts. Then, Gertrude and Bob swap their gifts. Then, Claire and Alice swap their gifts. Then, Alice and Claire swap their gifts. Then, Fred and Bob swap their gifts. Then, Bob and Fred swap their gifts. Then, Alice and Eve swap their gifts. Then, Eve and Gertrude swap their gifts. Then, Gertrude and Eve swap their gifts. Then, Eve and Alice swap their gifts. Then, Dave and Alice swap their gifts. Then, Dave and Alice discuss something. Then, Dave and Alice swap their gifts. Then, Eve and Fred swap their gifts. Then, Fred and Eve swap their gifts. Then, Eve and Dave swap their gifts. Then, Dave and Eve swap their gifts. Then, nothing happens for a while. Then, Claire and Eve swap their gifts. Then, Claire and Eve discuss something. Then, Claire and Eve swap their gifts. Then, Fred and Eve swap their gifts. Then, Eve and Alice swap their gifts. Then, Alice and Eve swap their gifts. Then, Eve and Fred swap their gifts. Then, Gertrude and Eve swap their gifts. Then, Gertrude and Eve discuss something. Then, Gertrude and Eve swap their gifts. Then, Alice and Claire swap their gifts. Then, Claire and Alice swap their gifts. Then, nothing happens for a while. Then, Dave and Claire swap their gifts. Then, Claire and Dave swap their gifts. Then, Dave and Gertrude swap their gifts. Then, Dave and Gertrude discuss something. Then, Dave and Gertrude swap their gifts. Then, Alice and Claire swap their gifts. Then, Bob and Alice swap their gifts. Then, Alice and Bob swap their gifts. Then, nothing happens for a while. Then, Dave and Fred swap their gifts. Then, Dave and Fred discuss something. Then, Dave and Fred swap their gifts. Then, Alice and Claire swap their gifts. Then, Claire and Alice swap their gifts. Then, nothing happens for a while. Then, Alice and Bob swap their gifts. Then, Alice and Bob discuss something. Then, Alice and Bob swap their gifts. Then, Fred and Eve swap their gifts. Then, Eve and Claire swap their gifts. Then, Claire and Eve swap their gifts. Then, Eve and Fred swap their gifts. Then, Fred and Claire swap their gifts. Then, Claire and Fred swap their gifts. Then, Bob and Fred swap their gifts. Then, Fred and Bob swap their gifts. Then, Gertrude and Fred swap their gifts. Then, Fred and Gertrude swap their gifts. Then, Eve and Claire swap their gifts. Then, Eve and Claire discuss something. Then, Eve and Claire swap their gifts. Then, Dave and Eve swap their gifts. Then, Dave and Eve discuss something. Then, Dave and Eve swap their gifts. Then, nothing happens for a while. Then, Dave and Alice swap their gifts. Then, Alice and Bob swap their gifts. Then, Bob and Alice swap their gifts. Then, Alice and Dave swap their gifts. Then, nothing happens for a while. Then, Alice and Eve swap their gifts. Then, Eve and Fred swap their gifts. Then, Fred and Eve swap their gifts. Then, Eve and Alice swap their gifts. Then, Dave and Eve swap their gifts. Then, Eve and Dave swap their gifts. Then, Alice and Dave swap their gifts. Then, Alice and Dave discuss something. Then, Alice and Dave swap their gifts. Then, Fred and Alice swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Alice and Fred swap their gifts. Then, Claire and Alice swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Alice and Claire swap their gifts. Then, Bob and Eve swap their gifts. Then, Eve and Dave swap their gifts. Then, Dave and Eve swap their gifts. Then, Eve and Bob swap their gifts. Then, Gertrude and Fred swap their gifts. Then, Fred and Gertrude swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Gertrude and Dave swap their gifts. Then, Dave and Alice swap their gifts. Then, Alice and Dave swap their gifts. Then, Dave and Gertrude swap their gifts. Then, Alice and Claire swap their gifts. Then, Claire and Fred swap their gifts. Then, Fred and Claire swap their gifts. Then, Claire and Alice swap their gifts. Then, Dave and Alice swap their gifts. Then, Alice and Dave swap their gifts. Then, Claire and Fred swap their gifts. Then, Fred and Alice swap their gifts. Then, Alice and Fred swap their gifts. Then, Fred and Claire swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Bob and Dave swap their gifts. Then, Dave and Fred swap their gifts. Then, Fred and Dave swap their gifts. Then, Dave and Bob swap their gifts. Then, Claire and Dave swap their gifts. Then, Dave and Bob swap their gifts. Then, Bob and Dave swap their gifts. Then, Dave and Claire swap their gifts. Then, Dave and Alice swap their gifts. Then, Gertrude and Eve swap their gifts. Finally, Claire and Gertrude swap their gifts. At the end of the event, Bob has the\\nOptions:\\n(A) blue present\\n(B) green present\\n(C) brown present\\n(D) purple present\\n(E) orange ball\\n(F) pink ball\\n(G) red present\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}