{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1aabe1ef-b755-54bd-86fd-e7e35020d15a","task_key":"tracking~5fshuffled~5fobjects~5fseven~5fobjects--main--1aabe1ef-b755-54bd-86fd-e7e35020d15a","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are dancers at a square dance.…","config":"tracking_shuffled_objects_seven_objects","split":"main","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are dancers at a square dance. At the start of a song, they each have a partner: Alice is dancing with Melissa, Bob is dancing with Lola, Claire is dancing with Patrick, Dave is dancing with Sam, Eve is dancing with Izzi, Fred is dancing with Helga, and Gertrude is dancing with Rodrigo.\\nThroughout the song, the dancers often trade partners. First, Fred and Gertrude switch partners. Then, Claire and Fred switch partners. Then, Dave and Alice switch partners. Then, Alice and Bob switch partners. Then, Gertrude and Eve switch partners. Then, Dave and Gertrude switch partners. Finally, Alice and Dave switch partners. At the end of the dance, Dave is dancing with\\nOptions:\\n(A) Melissa\\n(B) Lola\\n(C) Patrick\\n(D) Sam\\n(E) Izzi\\n(F) Helga\\n(G) Rodrigo\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}