{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"dc800e99-b215-5b6e-954e-2a6cd6752ab7","task_key":"reasoning~5fabout~5fcolored~5fobjects--main--dc800e99-b215-5b6e-954e-2a6cd6752ab7","task_revision_id":"1","upstream_id":"","short_description":"On the desk, I see two silver booklets, one yellow sheet of paper, one silver…","config":"reasoning_about_colored_objects","split":"main","body":"{\"input\":\"On the desk, I see two silver booklets, one yellow sheet of paper, one silver pair of sunglasses, two yellow booklets, one yellow pair of sunglasses, three purple booklets, one purple sheet of paper, and two silver sheets of paper. If I remove all the silver items from the desk, how many pairs of sunglasses remain on it?\\nOptions:\\n(A) zero\\n(B) one\\n(C) two\\n(D) three\\n(E) four\\n(F) five\\n(G) six\\n(H) seven\\n(I) eight\\n(J) nine\\n(K) ten\\n(L) eleven\\n(M) twelve\\n(N) thirteen\\n(O) fourteen\\n(P) fifteen\\n(Q) sixteen\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}