{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8b8a462f-bb0d-5d0a-a8a3-b6f0701781fc","task_key":"bbeh~5fshuffled~5fobjects--full--8b8a462f-bb0d-5d0a-a8a3-b6f0701781fc","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are dancers at a square dance.…","config":"bbeh_shuffled_objects","split":"full","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are dancers at a square dance. At the start of a song, they each have a partner: Alice is dancing with Lola, Bob is dancing with Ophelia, Claire is dancing with Sam, Dave is dancing with Helga, Eve is dancing with Patrick, Fred is dancing with Jamie, and Gertrude is dancing with Rodrigo.\\nThroughout the song, the dancers often trade partners. First, Fred and Bob switch partners. Then, Alice and Fred switch partners. Then, Fred and Gertrude switch partners. Then, Gertrude and Fred switch partners. Then, Fred and Alice switch partners. Then, Dave and Fred switch partners. Then, Fred and Eve switch partners. Then, Eve and Fred switch partners. Then, Fred and Dave switch partners. Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Claire switch partners. Then, Claire and Alice switch partners. Then, nothing happens for a while. Then, Eve and Alice switch partners. Then, Alice and Eve switch partners. Then, Eve and Gertrude switch partners. Then, Gertrude and Dave switch partners. Then, Dave and Gertrude switch partners. Then, Gertrude and Eve switch partners. Then, nothing happens for a while. Then, nothing happens for a while. Then, Claire and Gertrude switch partners. Then, Gertrude and Claire switch partners. Then, Fred and Alice switch partners. Then, Alice and Dave switch partners. Then, Dave and Alice switch partners. Then, Alice and Fred switch partners. Then, Alice and Bob switch partners. Then, Bob and Alice switch partners. Then, Bob and Claire switch partners. Then, Claire and Bob switch partners. Then, nothing happens for a while. Then, Claire and Gertrude switch partners. Then, Gertrude and Claire switch partners. Then, Alice and Fred switch partners. Then, Fred and Alice switch partners. Then, Claire and Gertrude switch partners. Then, Gertrude and Claire switch partners. Then, Alice and Claire switch partners. Then, Alice and Claire discuss something. Then, Alice and Claire switch partners. Then, Bob and Claire switch partners. Then, Claire and Eve switch partners. Then, Eve and Claire switch partners. Then, Claire and Bob switch partners. Then, Claire and Dave switch partners. Then, Dave and Fred switch partners. Then, Fred and Dave switch partners. Then, Dave and Claire switch partners. Then, Claire and Bob switch partners. Then, Bob and Claire switch partners. Then, nothing happens for a while. Then, nothing happens for a while. Then, Eve and Alice switch partners. Then, Alice and Eve switch partners. Then, Alice and Claire switch partners. Then, Claire and Bob switch partners. Then, Bob and Claire switch partners. Then, Claire and Alice switch partners. Then, Claire and Alice switch partners. Then, Alice and Eve switch partners. Then, Eve and Alice switch partners. Then, Alice and Claire switch partners. Then, Gertrude and Alice switch partners. Then, Gertrude and Alice discuss something. Then, Gertrude and Alice switch partners. Then, Alice and Fred switch partners. Then, Fred and Alice switch partners. Then, Eve and Claire switch partners. Then, Claire and Bob switch partners. Then, Bob and Claire switch partners. Then, Claire and Eve switch partners. Then, Alice and Fred switch partners. Then, Alice and Fred discuss something. Then, Alice and Fred switch partners. Then, Alice and Bob switch partners. Then, Alice and Bob discuss something. Then, Alice and Bob switch partners. Then, Alice and Gertrude switch partners. Then, Gertrude and Dave switch partners. Then, Dave and Gertrude switch partners. Then, Gertrude and Alice switch partners. Then, Eve and Fred switch partners. Then, Fred and Dave switch partners. Then, Dave and Fred switch partners. Then, Fred and Eve switch partners. Then, Gertrude and Dave switch partners. Then, Dave and Gertrude switch partners. Then, Eve and Alice switch partners. Then, Claire and Alice switch partners. Then, Alice and Dave switch partners. Then, Dave and Alice switch partners. Then, Alice and Claire switch partners. Then, Alice and Claire switch partners. Then, Claire and Alice switch partners. Then, Dave and Alice switch partners. Then, Alice and Eve switch partners. Then, Eve and Alice switch partners. Then, Alice and Dave switch partners. Then, nothing happens for a while. Then, Fred and Dave switch partners. Then, Fred and Dave discuss something. Then, Fred and Dave switch partners. Then, Fred and Claire switch partners. Then, Claire and Eve switch partners. Then, Eve and Claire switch partners. Then, Claire and Fred switch partners. Then, Claire and Fred switch partners. Then, Claire and Fred discuss something. Then, Claire and Fred switch partners. Then, Fred and Gertrude switch partners. Then, Gertrude and Fred switch partners. Then, nothing happens for a while. Then, Alice and Bob switch partners. Then, Bob and Fred switch partners. Then, Fred and Bob switch partners. Then, Bob and Alice switch partners. Then, Eve and Bob switch partners. Then, Bob and Eve switch partners. Then, Alice and Eve switch partners. Then, Alice and Eve discuss something. Then, Alice and Eve switch partners. Then, Eve and Gertrude switch partners. Then, Eve and Gertrude discuss something. Then, Eve and Gertrude switch partners. Then, Bob and Dave switch partners. Then, Dave and Bob switch partners. Then, Alice and Gertrude switch partners. Then, Gertrude and Eve switch partners. Then, Eve and Gertrude switch partners. Then, Gertrude and Alice switch partners. Then, Gertrude and Dave switch partners. Then, Dave and Eve switch partners. Then, Eve and Dave switch partners. Then, Dave and Gertrude switch partners. Then, nothing happens for a while. Then, Gertrude and Alice switch partners. Then, Alice and Dave switch partners. Then, Dave and Alice switch partners. Then, Alice and Gertrude switch partners. Then, Alice and Fred switch partners. Then, Fred and Dave switch partners. Then, Dave and Fred switch partners. Then, Fred and Alice switch partners. Then, Claire and Alice switch partners. Then, Alice and Fred switch partners. Then, Fred and Alice switch partners. Then, Alice and Claire switch partners. Then, nothing happens for a while. Then, Fred and Bob switch partners. Then, Fred and Bob discuss something. Then, Fred and Bob switch partners. Then, Eve and Bob switch partners. Then, Bob and Claire switch partners. Then, Claire and Bob switch partners. Then, Bob and Eve switch partners. Then, nothing happens for a while. Then, Eve and Alice switch partners. Then, Eve and Alice discuss something. Then, Eve and Alice switch partners. Then, nothing happens for a while. Then, Eve and Gertrude switch partners. Then, Gertrude and Eve switch partners. Then, Bob and Alice switch partners. Then, Alice and Bob switch partners. Then, Claire and Dave switch partners. Then, Dave and Bob switch partners. Then, Bob and Dave switch partners. Then, Dave and Claire switch partners. Then, Dave and Eve switch partners. Then, Eve and Claire switch partners. Then, Claire and Eve switch partners. Then, Eve and Dave switch partners. Then, Dave and Bob switch partners. Then, Bob and Alice switch partners. Then, Alice and Bob switch partners. Then, Bob and Dave switch partners. Then, Fred and Eve switch partners. Then, Eve and Fred switch partners. Then, Eve and Bob switch partners. Then, Bob and Eve switch partners. Then, Dave and Bob switch partners. Then, Bob and Alice switch partners. Then, Alice and Bob switch partners. Then, Bob and Dave switch partners. Then, Gertrude and Claire switch partners. Then, Gertrude and Claire discuss something. Then, Gertrude and Claire switch partners. Then, nothing happens for a while. Then, nothing happens for a while. Then, Claire and Bob switch partners. Then, Bob and Claire switch partners. Then, nothing happens for a while. Then, Dave and Bob switch partners. Then, Dave and Bob discuss something. Then, Dave and Bob switch partners. Then, nothing happens for a while. Then, Fred and Eve switch partners. Then, Eve and Gertrude switch partners. Then, Gertrude and Dave switch partners. Then, Dave and Gertrude switch partners. Then, Gertrude and Eve switch partners. Then, nothing happens for a while. Then, Eve and Alice switch partners. Then, Alice and Fred switch partners. Then, Fred and Alice switch partners. Then, Alice and Eve switch partners. Then, Gertrude and Bob switch partners. Then, Gertrude and Bob discuss something. Then, Gertrude and Bob switch partners. Then, Gertrude and Claire switch partners. Then, Claire and Gertrude switch partners. Then, Alice and Fred switch partners. Then, Fred and Alice switch partners. Then, nothing happens for a while. Then, Eve and Alice switch partners. Then, Alice and Gertrude switch partners. Then, Gertrude and Alice switch partners. Then, Alice and Eve switch partners. Then, Dave and Gertrude switch partners. Then, Gertrude and Alice switch partners. Then, Alice and Gertrude switch partners. Then, Gertrude and Dave switch partners. Then, nothing happens for a while. Then, nothing happens for a while. Then, Bob and Gertrude switch partners. Then, Bob and Gertrude discuss something. Then, Bob and Gertrude switch partners. Then, Dave and Alice switch partners. Then, Dave and Alice discuss something. Then, Dave and Alice switch partners. Then, Fred and Dave switch partners. Then, Dave and Alice switch partners. Then, Alice and Dave switch partners. Then, Dave and Fred switch partners. Then, Dave and Alice switch partners. Then, Alice and Claire switch partners. Then, Claire and Alice switch partners. Then, Alice and Dave switch partners. Then, Gertrude and Dave switch partners. Then, Gertrude and Dave discuss something. Then, Gertrude and Dave switch partners. Then, Fred and Bob switch partners. Then, Bob and Fred switch partners. Then, nothing happens for a while. Then, Alice and Bob switch partners. Then, Bob and Dave switch partners. Then, Dave and Bob switch partners. Then, Bob and Alice switch partners. Then, nothing happens for a while. Then, Bob and Dave switch partners. Then, Bob and Dave discuss something. Then, Bob and Dave switch partners. Then, Bob and Eve switch partners. Then, Bob and Eve discuss something. Then, Bob and Eve switch partners. Then, nothing happens for a while. Then, Eve and Gertrude switch partners. Then, Eve and Gertrude discuss something. Then, Eve and Gertrude switch partners. Then, Gertrude and Alice switch partners. Then, Gertrude and Alice discuss something. Then, Gertrude and Alice switch partners. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Claire and Gertrude switch partners. Then, Gertrude and Alice switch partners. Then, Alice and Gertrude switch partners. Then, Gertrude and Claire switch partners. Then, Bob and Eve switch partners. Then, Eve and Gertrude switch partners. Then, Gertrude and Eve switch partners. Then, Eve and Bob switch partners. Then, Bob and Gertrude switch partners. Then, Bob and Gertrude discuss something. Then, Bob and Gertrude switch partners. Then, Bob and Claire switch partners. Then, Claire and Dave switch partners. Then, Dave and Claire switch partners. Then, Claire and Bob switch partners. Then, Dave and Claire switch partners. Then, Claire and Dave switch partners. Then, Alice and Fred switch partners. Then, Fred and Alice switch partners. Then, nothing happens for a while. Then, Dave and Fred switch partners. Then, Dave and Fred discuss something. Then, Dave and Fred switch partners. Then, Alice and Claire switch partners. Then, Claire and Alice switch partners. Then, Dave and Claire switch partners. Then, Dave and Claire discuss something. Then, Dave and Claire switch partners. Then, Claire and Fred switch partners. Then, Fred and Claire switch partners. Then, Claire and Fred switch partners. Then, nothing happens for a while. Then, Bob and Alice switch partners. Then, Bob and Alice discuss something. Then, Bob and Alice switch partners. Then, nothing happens for a while. Then, nothing happens for a while. Then, nothing happens for a while. Then, Alice and Fred switch partners. Then, Fred and Gertrude switch partners. Then, Gertrude and Fred switch partners. Then, Fred and Alice switch partners. Then, Dave and Gertrude switch partners. Then, Gertrude and Dave switch partners. Then, Gertrude and Claire switch partners. Then, Gertrude and Claire discuss something. Then, Gertrude and Claire switch partners. Then, Bob and Dave switch partners. Then, Dave and Bob switch partners. Then, Alice and Eve switch partners. Then, Eve and Claire switch partners. Then, Claire and Eve switch partners. Then, Eve and Alice switch partners. Then, Fred and Claire switch partners. Then, Fred and Claire discuss something. Then, Fred and Claire switch partners. Then, nothing happens for a while. Then, nothing happens for a while. Then, Claire and Fred switch partners. Then, Claire and Fred discuss something. Then, Claire and Fred switch partners. Then, Gertrude and Eve switch partners. Then, Gertrude and Eve discuss something. Then, Gertrude and Eve switch partners. Then, Dave and Fred switch partners. Then, Fred and Gertrude switch partners. Then, Gertrude and Fred switch partners. Then, Fred and Dave switch partners. Then, Eve and Bob switch partners. Then, Bob and Claire switch partners. Then, Claire and Bob switch partners. Then, Bob and Eve switch partners. Then, nothing happens for a while. Then, Gertrude and Bob switch partners. Then, Gertrude and Bob discuss something. Then, Gertrude and Bob switch partners. Then, Fred and Dave switch partners. Then, Dave and Claire switch partners. Then, Claire and Dave switch partners. Then, Dave and Fred switch partners. Then, Eve and Gertrude switch partners. Then, Gertrude and Eve switch partners. Then, Gertrude and Eve switch partners. Then, Eve and Gertrude switch partners. Then, Eve and Dave switch partners. Then, Dave and Fred switch partners. Then, Fred and Dave switch partners. Then, Dave and Eve switch partners. Then, Bob and Dave switch partners. Then, Dave and Bob switch partners. Then, Claire and Alice switch partners. Then, Alice and Eve switch partners. Then, Eve and Alice switch partners. Then, Alice and Claire switch partners. Then, Eve and Fred switch partners. Then, Eve and Fred discuss something. Then, Eve and Fred switch partners. Then, Gertrude and Dave switch partners. Then, Dave and Gertrude switch partners. Then, Dave and Bob switch partners. Then, Bob and Alice switch partners. Then, Alice and Bob switch partners. Then, Bob and Dave switch partners. Then, Gertrude and Alice switch partners. Then, Alice and Bob switch partners. Finally, Dave and Fred switch partners. At the end of the dance, Dave is dancing with\\nOptions:\\n(A) Lola\\n(B) Ophelia\\n(C) Sam\\n(D) Helga\\n(E) Patrick\\n(F) Jamie\\n(G) Rodrigo\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}