{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4d7e7468-7dd0-5b5d-9ff6-5df4c0286cbb","task_key":"tracking~5fshuffled~5fobjects~5fseven~5fobjects--main--4d7e7468-7dd0-5b5d-9ff6-5df4c0286cbb","task_revision_id":"1","upstream_id":"","short_description":"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are holding a white elephant…","config":"tracking_shuffled_objects_seven_objects","split":"main","body":"{\"input\":\"Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are holding a white elephant gift exchange. At the start of the event, they are each holding a present of a different color: Alice has a pink ball, Bob has a brown present, Claire has a red present, Dave has a blue present, Eve has a white present, Fred has a orange ball, and Gertrude has a black ball.\\nAs the event progresses, pairs of people swap gifts. First, Alice and Dave swap their gifts. Then, Claire and Dave swap their gifts. Then, Fred and Gertrude swap their gifts. Then, Eve and Gertrude swap their gifts. Then, Eve and Alice swap their gifts. Then, Dave and Bob swap their gifts. Finally, Gertrude and Dave swap their gifts. At the end of the event, Alice has the\\nOptions:\\n(A) pink ball\\n(B) brown present\\n(C) red present\\n(D) blue present\\n(E) white present\\n(F) orange ball\\n(G) black ball\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}