{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"54ab6f26-8393-5d56-bd81-5c80e8e15888","task_key":"reasoning~5fabout~5fcolored~5fobjects--main--54ab6f26-8393-5d56-bd81-5c80e8e15888","task_revision_id":"1","upstream_id":"","short_description":"On the desk, you see three silver pairs of sunglasses, two teal envelopes, three…","config":"reasoning_about_colored_objects","split":"main","body":"{\"input\":\"On the desk, you see three silver pairs of sunglasses, two teal envelopes, three pink pairs of sunglasses, one silver envelope, two pink envelopes, three teal pairs of sunglasses, two teal scrunchiephone chargers, and three pink scrunchiephone chargers. If I remove all the envelopes from the desk, how many pink items remain on it?\\nOptions:\\n(A) zero\\n(B) one\\n(C) two\\n(D) three\\n(E) four\\n(F) five\\n(G) six\\n(H) seven\\n(I) eight\\n(J) nine\\n(K) ten\\n(L) eleven\\n(M) twelve\\n(N) thirteen\\n(O) fourteen\\n(P) fifteen\\n(Q) sixteen\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}