{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"18a0b381-c0be-5bc9-95a9-1c74bf03b22d","task_key":"reasoning~5fabout~5fcolored~5fobjects--main--18a0b381-c0be-5bc9-95a9-1c74bf03b22d","task_revision_id":"1","upstream_id":"","short_description":"On the desk, there are three black sheets of paper, two black envelopes, one…","config":"reasoning_about_colored_objects","split":"main","body":"{\"input\":\"On the desk, there are three black sheets of paper, two black envelopes, one brown cup, one brown stress ball, one brown envelope, one brown sheet of paper, and one black cup. If I remove all the brown objects from the desk, how many envelopes remain on it?\\nOptions:\\n(A) zero\\n(B) one\\n(C) two\\n(D) three\\n(E) four\\n(F) five\\n(G) six\\n(H) seven\\n(I) eight\\n(J) nine\\n(K) ten\\n(L) eleven\\n(M) twelve\\n(N) thirteen\\n(O) fourteen\\n(P) fifteen\\n(Q) sixteen\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}