{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1887e933-6e5f-53b2-b8f5-8c3631224109","task_key":"bbeh~5fshuffled~5fobjects--full--1887e933-6e5f-53b2-b8f5-8c3631224109","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are friends and avid readers…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are friends and avid readers who occasionally trade books. At the start of the semester, they each buy one new book: Alice gets The Pearl, Bob gets The Great Gatsby, Claire gets Lolita, Dave gets Ulysses, Eve gets Frankenstein, Fred gets Moby Dick, and Gertrude gets The Odyssey.\\nAs the semester proceeds, they start trading around the new books. First, Gertrude and Fred swap books. Then, Fred and Gertrude swap books. Then, Fred and Gertrude discuss something. Then, Fred and Gertrude swap books. Then, Claire and Eve swap books. Then, Eve and Fred swap books. Then, Fred and Eve swap books. Then, Eve and Claire swap books. Then, Bob and Alice swap books. Then, Bob and Alice discuss something. Then, Bob and Alice swap books. Then, Fred and Dave swap books. Then, Dave and Gertrude swap books. Then, Gertrude and Dave swap books. Then, Dave and Fred swap books. Then, nothing happens for a while. Then, Gertrude and Alice swap books. Then, Alice and Dave swap books. Then, Dave and Alice swap books. Then, Alice and Gertrude swap books. Then, Gertrude and Claire swap books. Then, Gertrude and Claire discuss something. Then, Gertrude and Claire swap books. Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Gertrude swap books. Then, Dave and Gertrude discuss something. Then, Dave and Gertrude swap books. Then, nothing happens for a while. Then, nothing happens for a while. Then, Gertrude and Claire swap books. Then, Gertrude and Claire discuss something. Then, Gertrude and Claire swap books. Then, Eve and Alice swap books. Then, Alice and Eve swap books. Then, Fred and Alice swap books. Then, Fred and Alice discuss something. Then, Fred and Alice swap books. Then, nothing happens for a while. Then, Dave and Fred swap books. Then, Dave and Fred discuss something. Then, Dave and Fred swap books. Then, Alice and Gertrude swap books. Then, Gertrude and Dave swap books. Then, Dave and Gertrude swap books. Then, Gertrude and Alice swap books. Then, Dave and Bob swap books. Then, Bob and Gertrude swap books. Then, Gertrude and Bob swap books. Then, Bob and Dave swap books. Then, nothing happens for a while. Then, Claire and Dave swap books. Then, Claire and Dave discuss something. Then, Claire and Dave swap books. Then, nothing happens for a while. Then, Dave and Bob swap books. Then, Dave and Bob discuss something. Then, Dave and Bob swap books. Then, nothing happens for a while. Then, Claire and Bob swap books. Then, Bob and Claire swap books. Then, Gertrude and Eve swap books. Then, Eve and Gertrude swap books. Then, Bob and Dave swap books. Then, nothing happens for a while. Then, Bob and Gertrude swap books. Then, Gertrude and Dave swap books. Then, Dave and Gertrude swap books. Then, Gertrude and Bob swap books. Then, Alice and Gertrude swap books. Then, Gertrude and Alice swap books. Then, Alice and Bob swap books. Then, Alice and Bob discuss something. Then, Alice and Bob swap books. Then, Bob and Dave swap books. Then, Bob and Dave discuss something. Then, Bob and Dave swap books. Then, Gertrude and Fred swap books. Then, Gertrude and Fred discuss something. Then, Gertrude and Fred swap books. Then, Gertrude and Dave swap books. Then, Dave and Gertrude swap books. Then, Eve and Claire swap books. Then, Claire and Eve swap books. Then, Bob and Eve swap books. Then, Eve and Dave swap books. Then, Dave and Eve swap books. Then, Eve and Bob swap books. Then, nothing happens for a while. Then, Claire and Eve swap books. Then, Eve and Bob swap books. Then, Bob and Eve swap books. Then, Eve and Claire swap books. Then, Claire and Eve swap books. Then, Eve and Dave swap books. Then, Dave and Eve swap books. Then, Eve and Claire swap books. Then, Eve and Fred swap books. Then, Fred and Dave swap books. Then, Dave and Fred swap books. Then, Fred and Eve swap books. Then, nothing happens for a while. Then, Gertrude and Eve swap books. Then, Gertrude and Eve discuss something. Then, Gertrude and Eve swap books. Then, Fred and Bob swap books. Then, Bob and Fred swap books. Then, Alice and Eve swap books. Then, Eve and Fred swap books. Then, Fred and Eve swap books. Then, Eve and Alice swap books. Then, Gertrude and Fred swap books. Then, Fred and Gertrude swap books. Then, nothing happens for a while. Then, nothing happens for a while. Then, Gertrude and Fred swap books. Then, Fred and Gertrude swap books. Then, nothing happens for a while. Then, Claire and Eve swap books. Then, Claire and Eve discuss something. Then, Claire and Eve swap books. Then, Eve and Gertrude swap books. Then, Eve and Gertrude discuss something. Then, Eve and Gertrude swap books. Then, Gertrude and Eve swap books. Then, Gertrude and Eve discuss something. Then, Gertrude and Eve swap books. Then, Dave and Fred swap books. Then, Dave and Fred discuss something. Then, Dave and Fred swap books. Then, nothing happens for a while. Then, Fred and Claire swap books. Then, Fred and Claire discuss something. Then, Fred and Claire swap books. Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Fred swap books. Then, Alice and Fred discuss something. Then, Alice and Fred swap books. Then, nothing happens for a while. Then, Eve and Gertrude swap books. Then, Eve and Gertrude discuss something. Then, Eve and Gertrude swap books. Then, Dave and Gertrude swap books. Then, Gertrude and Dave swap books. Then, Alice and Eve swap books. Then, Eve and Alice swap books. Then, Eve and Alice swap books. Then, Alice and Fred swap books. Then, Fred and Alice swap books. Then, Alice and Eve swap books. Then, nothing happens for a while. Then, Eve and Alice swap books. Then, Gertrude and Bob swap books. Then, Bob and Fred swap books. Then, Fred and Bob swap books. Then, Bob and Gertrude swap books. Then, Alice and Gertrude swap books. Then, Alice and Gertrude discuss something. Then, Alice and Gertrude swap books. Then, Dave and Bob swap books. Then, Bob and Claire swap books. Then, Claire and Bob swap books. Then, Bob and Dave swap books. Then, nothing happens for a while. Then, Dave and Fred swap books. Then, Fred and Dave swap books. Then, Gertrude and Eve swap books. Then, Eve and Dave swap books. Then, Dave and Eve swap books. Then, Eve and Gertrude swap books. Then, nothing happens for a while. Then, Claire and Dave swap books. Then, Dave and Claire swap books. Then, Gertrude and Alice swap books. Then, Alice and Gertrude swap books. Then, Fred and Eve swap books. Then, Eve and Claire swap books. Then, Claire and Eve swap books. Then, Eve and Fred swap books. Then, nothing happens for a while. Then, Dave and Bob swap books. Then, Bob and Eve swap books. Then, Eve and Bob swap books. Then, Bob and Dave swap books. Then, Claire and Fred swap books. Then, Fred and Gertrude swap books. Then, Gertrude and Fred swap books. Then, Fred and Claire swap books. Then, Fred and Dave swap books. Then, Dave and Fred swap books. Then, Dave and Fred swap books. Then, Dave and Fred discuss something. Then, Dave and Fred swap books. Then, Bob and Fred swap books. Then, Bob and Fred discuss something. Then, Bob and Fred swap books. Then, Eve and Dave swap books. Then, Dave and Eve swap books. Then, Alice and Claire swap books. Then, Claire and Alice swap books. Then, Fred and Gertrude swap books. Then, Fred and Gertrude discuss something. Then, Fred and Gertrude swap books. Then, Claire and Fred swap books. Then, Fred and Claire swap books. Then, Alice and Eve swap books. Then, Eve and Claire swap books. Then, Claire and Eve swap books. Then, Eve and Alice swap books. Then, Eve and Fred swap books. Then, Fred and Claire swap books. Then, Claire and Fred swap books. Then, Fred and Eve swap books. Then, Dave and Eve swap books. Then, Dave and Eve discuss something. Then, Dave and Eve swap books. Then, Claire and Bob swap books. Then, Bob and Claire swap books. Then, nothing happens for a while. Then, Gertrude and Fred swap books. Then, Fred and Bob swap books. Then, Bob and Fred swap books. Then, Fred and Gertrude swap books. Then, Gertrude and Eve swap books. Then, Eve and Alice swap books. Then, Alice and Eve swap books. Then, Eve and Gertrude swap books. Then, Alice and Claire swap books. Then, Alice and Claire discuss something. Then, Alice and Claire swap books. Then, nothing happens for a while. Then, Eve and Gertrude swap books. Then, Gertrude and Fred swap books. Then, Fred and Gertrude swap books. Then, Gertrude and Eve swap books. Then, Dave and Claire swap books. Then, Claire and Fred swap books. Then, Fred and Claire swap books. Then, Claire and Dave swap books. Then, Gertrude and Alice swap books. Then, Alice and Gertrude swap books. Then, Alice and Dave swap books. Then, Dave and Bob swap books. Then, Bob and Dave swap books. Then, Dave and Alice swap books. Then, Fred and Claire swap books. Then, Claire and Fred swap books. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Eve and Bob swap books. Then, Bob and Eve swap books. Then, nothing happens for a while. Then, nothing happens for a while. Then, Dave and Alice swap books. Then, Alice and Dave swap books. Then, Bob and Fred swap books. Then, Fred and Eve swap books. Then, Eve and Fred swap books. Then, nothing happens for a while. Then, Bob and Dave swap books. Then, Dave and Alice swap books. Then, Alice and Dave swap books. Then, Dave and Bob swap books. Then, Fred and Dave swap books. Then, Fred and Dave discuss something. Then, Fred and Dave swap books. Then, nothing happens for a while. Then, Eve and Bob swap books. Then, Bob and Alice swap books. Then, Alice and Bob swap books. Then, Bob and Eve swap books. Then, Fred and Claire swap books. Then, Claire and Dave swap books. Then, Dave and Claire swap books. Then, Claire and Fred swap books. Then, nothing happens for a while. Then, nothing happens for a while. Then, Gertrude and Fred swap books. Then, Gertrude and Fred discuss something. Then, Gertrude and Fred swap books. Then, nothing happens for a while. Then, Alice and Claire swap books. Then, Claire and Alice swap books. Then, Eve and Alice swap books. Then, Eve and Alice discuss something. Then, Eve and Alice swap books. Then, Alice and Gertrude swap books. Then, Gertrude and Dave swap books. Then, Dave and Gertrude swap books. Then, Gertrude and Alice swap books. Then, Bob and Claire swap books. Then, Bob and Claire discuss something. Then, Bob and Claire swap books. Then, Dave and Fred swap books. Then, Fred and Dave swap books. Then, Alice and Gertrude swap books. Then, Gertrude and Fred swap books. Then, Fred and Gertrude swap books. Then, Gertrude and Alice swap books. Then, Claire and Dave swap books. Then, Claire and Dave discuss something. Then, Claire and Dave swap books. Then, Fred and Gertrude swap books. Then, Gertrude and Dave swap books. Then, Dave and Gertrude swap books. Then, Gertrude and Fred swap books. Then, Eve and Claire swap books. Then, Claire and Bob swap books. Then, Bob and Claire swap books. Then, Claire and Eve swap books. Then, Eve and Claire swap books. Then, Claire and Eve swap books. Then, Gertrude and Dave swap books. Then, Dave and Gertrude swap books. Then, Alice and Fred swap books. Then, Fred and Alice swap books. Then, Alice and Claire swap books. Then, Claire and Alice swap books. Then, nothing happens for a while. Then, Fred and Alice swap books. Then, Fred and Alice discuss something. Then, Fred and Alice swap books. Then, Fred and Bob swap books. Then, Bob and Fred swap books. Then, Fred and Eve swap books. Then, Eve and Fred swap books. Then, Claire and Gertrude swap books. Then, Gertrude and Claire swap books. Then, nothing happens for a while. Then, Claire and Gertrude swap books. Then, Gertrude and Alice swap books. Then, Alice and Gertrude swap books. Then, Gertrude and Claire swap books. Then, Alice and Gertrude swap books. Then, Alice and Gertrude discuss something. Then, Alice and Gertrude swap books. Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Fred swap books. Then, Fred and Alice swap books. Then, Dave and Alice swap books. Then, Alice and Gertrude swap books. Then, Gertrude and Alice swap books. Then, Alice and Dave swap books. Then, Fred and Gertrude swap books. Then, Gertrude and Alice swap books. Then, Alice and Gertrude swap books. Then, Gertrude and Fred swap books. Then, Bob and Gertrude swap books. Then, Gertrude and Claire swap books. Then, Claire and Gertrude swap books. Then, Gertrude and Bob swap books. Then, nothing happens for a while. Then, Dave and Gertrude swap books. Then, Gertrude and Dave swap books. Then, Eve and Fred swap books. Then, Fred and Eve swap books. Then, Bob and Fred swap books. Then, Bob and Fred discuss something. Then, Bob and Fred swap books. Then, Fred and Dave swap books. Then, Dave and Gertrude swap books. Then, Gertrude and Dave swap books. Then, Dave and Fred swap books. Then, nothing happens for a while. Then, Dave and Eve swap books. Then, Claire and Gertrude swap books. Finally, Fred and Eve swap books. At the end of the semester, Dave has\\nOptions:\\n(A) The Pearl\\n(B) The Great Gatsby\\n(C) Lolita\\n(D) Ulysses\\n(E) Frankenstein\\n(F) Moby Dick\\n(G) The Odyssey\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}