{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1780fd0f-adc4-546f-b8b9-7eabcec808ac","task_key":"bbeh~5fshuffled~5fobjects--full--1780fd0f-adc4-546f-b8b9-7eabcec808ac","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are holding a white elephant…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a purple present, Bob has a orange ball, Claire has a white present, Dave has a green present, Eve has a yellow present, Fred has a brown present, and Gertrude has a red present.\\nAs the event progresses, pairs of people swap gifts. First, Eve and Bob swap their gifts. Then, Alice and Bob swap their gifts. Then, Bob and Alice swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Claire swap their gifts. Then, Claire and Fred swap their gifts. Then, Fred and Claire swap their gifts. Then, Claire and Alice swap their gifts. Then, nothing happens for a while. Then, Dave and Fred swap their gifts. Then, Fred and Dave swap their gifts. Then, nothing happens for a while. Then, Bob and Claire swap their gifts. Then, Claire and Eve swap their gifts. Then, Eve and Claire swap their gifts. Then, Claire and Bob swap their gifts. Then, Bob and Claire swap their gifts. Then, Claire and Dave swap their gifts. Then, Dave and Claire swap their gifts. Then, Claire and Bob swap their gifts. Then, Bob and Claire swap their gifts. Then, Claire and Eve swap their gifts. Then, Eve and Claire swap their gifts. Then, Claire and Bob swap their gifts. Then, Gertrude and Dave swap their gifts. Then, Gertrude and Dave discuss something. Then, Gertrude and Dave swap their gifts. Then, Bob and Fred swap their gifts. Then, Fred and Bob swap their gifts. Then, Dave and Bob swap their gifts. Then, Bob and Dave swap their gifts. Then, nothing happens for a while. Then, Eve and Dave swap their gifts. Then, Dave and Eve swap their gifts. Then, Gertrude and Eve swap their gifts. Then, Gertrude and Eve discuss something. Then, Gertrude and Eve swap their gifts. Then, Alice and Dave swap their gifts. Then, Alice and Dave discuss something. Then, Alice and Dave swap their gifts. Then, Bob and Eve swap their gifts. Then, Bob and Eve discuss something. Then, Bob and Eve swap their gifts. Then, nothing happens for a while. Then, Fred and Claire swap their gifts. Then, Claire and Fred swap their gifts. Then, Eve and Alice swap their gifts. Then, Alice and Eve swap their gifts. Then, Fred and Gertrude swap their gifts. Then, Gertrude and Fred swap their gifts. Then, Claire and Dave swap their gifts. Then, Dave and Fred swap their gifts. Then, Fred and Dave swap their gifts. Then, Dave and Claire swap their gifts. Then, Alice and Bob swap their gifts. Then, Alice and Bob discuss something. Then, Alice and Bob swap their gifts. Then, Eve and Dave swap their gifts. Then, Eve and Dave discuss something. Then, Eve and Dave swap their gifts. Then, Claire and Alice swap their gifts. Then, Alice and Bob swap their gifts. Then, Bob and Alice swap their gifts. Then, Alice and Claire swap their gifts. Then, nothing happens for a while. Then, Alice and Eve swap their gifts. Then, Eve and Alice swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Alice and Dave swap their gifts. Then, Alice and Dave discuss something. Then, Alice and Dave swap their gifts. Then, Claire and Bob swap their gifts. Then, Claire and Bob discuss something. Then, Claire and Bob swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Dave and Alice swap their gifts. Then, Dave and Alice discuss something. Then, Dave and Alice swap their gifts. Then, nothing happens for a while. Then, Eve and Bob swap their gifts. Then, Bob and Dave swap their gifts. Then, Dave and Bob swap their gifts. Then, Bob and Eve swap their gifts. Then, Gertrude and Claire swap their gifts. Then, Claire and Fred swap their gifts. Then, Fred and Claire swap their gifts. Then, Claire and Gertrude swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Eve swap their gifts. Then, Eve and Gertrude swap their gifts. Then, Gertrude and Alice swap their gifts. Then, nothing happens for a while. Then, Alice and Dave swap their gifts. Then, Dave and Bob swap their gifts. Then, Bob and Dave swap their gifts. Then, Dave and Alice swap their gifts. Then, Dave and Fred swap their gifts. Then, Dave and Fred discuss something. Then, Dave and Fred swap their gifts. Then, Dave and Claire swap their gifts. Then, Dave and Eve swap their gifts. Then, Eve and Dave swap their gifts. Then, Bob and Dave swap their gifts. Then, Dave and Alice swap their gifts. Then, Alice and Dave swap their gifts. Then, Dave and Bob swap their gifts. Then, Alice and Fred swap their gifts. Then, Fred and Eve swap their gifts. Then, Eve and Fred swap their gifts. Then, Fred and Alice swap their gifts. Then, nothing happens for a while. Then, Gertrude and Alice swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Bob swap their gifts. Then, Gertrude and Bob discuss something. Then, Gertrude and Bob swap their gifts. Then, Gertrude and Fred swap their gifts. Then, Gertrude and Fred discuss something. Then, Gertrude and Fred swap their gifts. Then, Gertrude and Fred swap their gifts. Then, Fred and Alice swap their gifts. Then, Alice and Fred swap their gifts. Then, Fred and Gertrude swap their gifts. Then, Fred and Gertrude swap their gifts. Then, Gertrude and Fred swap their gifts. Then, nothing happens for a while. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Alice swap their gifts. Then, Claire and Bob swap their gifts. Then, Claire and Bob discuss something. Then, Claire and Bob swap their gifts. Then, nothing happens for a while. Then, Eve and Gertrude swap their gifts. Then, Gertrude and Claire swap their gifts. Then, Claire and Gertrude swap their gifts. Then, Gertrude and Eve swap their gifts. Then, Gertrude and Dave swap their gifts. Then, Dave and Gertrude swap their gifts. Then, Gertrude and Eve swap their gifts. Then, Eve and Gertrude swap their gifts. Then, Alice and Fred swap their gifts. Then, Fred and Eve swap their gifts. Then, Eve and Fred swap their gifts. Then, Fred and Alice swap their gifts. Then, Claire and Gertrude swap their gifts. Then, Claire and Gertrude discuss something. Then, Claire and Gertrude swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Alice swap their gifts. Then, Dave and Alice discuss something. Then, Dave and Alice swap their gifts. Then, Alice and Bob swap their gifts. Then, Gertrude and Bob swap their gifts. Then, Bob and Eve swap their gifts. Then, Eve and Bob swap their gifts. Then, Bob and Gertrude swap their gifts. Then, Dave and Gertrude swap their gifts. Then, Gertrude and Dave swap their gifts. Then, Claire and Fred swap their gifts. Then, Claire and Fred discuss something. Then, Claire and Fred swap their gifts. Then, nothing happens for a while. Then, Gertrude and Fred swap their gifts. Then, Fred and Gertrude swap their gifts. Then, nothing happens for a while. Then, Claire and Eve swap their gifts. Then, Claire and Eve discuss something. Then, Claire and Eve swap their gifts. Then, Dave and Alice swap their gifts. Then, Alice and Dave swap their gifts. Then, Claire and Eve swap their gifts. Then, Claire and Eve discuss something. Then, Claire and Eve swap their gifts. Then, nothing happens for a while. Then, Gertrude and Eve swap their gifts. Then, Gertrude and Eve discuss something. Then, Gertrude and Eve swap their gifts. Then, Bob and Fred swap their gifts. Then, Fred and Bob swap their gifts. Then, Gertrude and Bob swap their gifts. Then, Bob and Gertrude swap their gifts. Then, nothing happens for a while. Then, Bob and Dave swap their gifts. Then, Bob and Dave discuss something. Then, Bob and Dave swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Gertrude swap their gifts. Then, Gertrude and Fred swap their gifts. Then, Fred and Gertrude swap their gifts. Then, Gertrude and Dave swap their gifts. Then, Dave and Eve swap their gifts. Then, Eve and Gertrude swap their gifts. Then, Gertrude and Eve swap their gifts. Then, Eve and Dave swap their gifts. Then, Bob and Fred swap their gifts. Then, Bob and Fred discuss something. Then, Bob and Fred swap their gifts. Then, Dave and Claire swap their gifts. Then, Claire and Dave swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Gertrude swap their gifts. Then, Eve and Bob swap their gifts. Then, Eve and Bob discuss something. Then, Eve and Bob swap their gifts. Then, Alice and Dave swap their gifts. Then, Dave and Gertrude swap their gifts. Then, Gertrude and Dave swap their gifts. Then, Dave and Alice swap their gifts. Then, nothing happens for a while. Then, Alice and Dave swap their gifts. Then, Alice and Dave discuss something. Then, Alice and Dave swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Claire swap their gifts. Then, Claire and Bob swap their gifts. Then, Bob and Claire swap their gifts. Then, Claire and Dave swap their gifts. Then, Dave and Fred swap their gifts. Then, Dave and Fred discuss something. Then, Dave and Fred swap their gifts. Then, nothing happens for a while. Then, Bob and Eve swap their gifts. Then, Eve and Claire swap their gifts. Then, Claire and Eve swap their gifts. Then, Eve and Bob swap their gifts. Then, Claire and Eve swap their gifts. Then, Eve and Claire swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, Fred and Eve swap their gifts. Then, Eve and Alice swap their gifts. Then, Alice and Eve swap their gifts. Then, Eve and Fred swap their gifts. Then, Claire and Dave swap their gifts. Then, Claire and Dave discuss something. Then, Claire and Dave swap their gifts. Then, Bob and Dave swap their gifts. Then, Dave and Bob swap their gifts. Then, nothing happens for a while. Then, Gertrude and Alice swap their gifts. Then, Alice and Gertrude swap their gifts. Then, Gertrude and Claire swap their gifts. Then, Claire and Gertrude swap their gifts. Then, Claire and Eve swap their gifts. Then, Eve and Claire swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, Claire and Fred swap their gifts. Then, Fred and Claire swap their gifts. Then, Fred and Alice swap their gifts. Then, Alice and Fred swap their gifts. Then, nothing happens for a while. Then, nothing happens for a while. Then, Bob and Claire swap their gifts. Then, Claire and Alice swap their gifts. Then, Alice and Claire swap their gifts. Then, Claire and Bob swap their gifts. Then, Gertrude and Claire swap their gifts. Then, Claire and Bob swap their gifts. Then, Bob and Claire swap their gifts. Then, Claire and Gertrude swap their gifts. Then, Claire and Eve swap their gifts. Then, Eve and Claire swap their gifts. Then, Fred and Eve swap their gifts. Then, Fred and Eve discuss something. Then, Fred and Eve swap their gifts. Then, Claire and Dave swap their gifts. Then, Claire and Dave discuss something. Then, Claire and Dave swap their gifts. Then, Bob and Alice swap their gifts. Then, Alice and Bob swap their gifts. Then, Claire and Eve swap their gifts. Then, Eve and Claire swap their gifts. Then, Gertrude and Bob swap their gifts. Then, Gertrude and Bob discuss something. Then, Gertrude and Bob swap their gifts. Then, Claire and Bob swap their gifts. Then, Dave and Fred swap their gifts. Finally, Bob and Eve swap their gifts. At the end of the event, Alice has the\\nOptions:\\n(A) purple present\\n(B) orange ball\\n(C) white present\\n(D) green present\\n(E) yellow present\\n(F) brown present\\n(G) red present\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}