{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"77172e90-c465-556a-a4dd-8e460a1cb6fd","task_key":"reasoning~5fabout~5fcolored~5fobjects--main--77172e90-c465-556a-a4dd-8e460a1cb6fd","task_revision_id":"1","upstream_id":"","short_description":"On the floor, there is one yellow textbook, one teal necklace, three yellow…","config":"reasoning_about_colored_objects","split":"main","body":"{\"input\":\"On the floor, there is one yellow textbook, one teal necklace, three yellow puzzles, three teal puzzles, two purple textbooks, two magenta pencils, one yellow pencil, two yellow necklaces, and one purple necklace. If I remove all the magenta things from the floor, how many puzzles remain on it?\\nOptions:\\n(A) zero\\n(B) one\\n(C) two\\n(D) three\\n(E) four\\n(F) five\\n(G) six\\n(H) seven\\n(I) eight\\n(J) nine\\n(K) ten\\n(L) eleven\\n(M) twelve\\n(N) thirteen\\n(O) fourteen\\n(P) fifteen\\n(Q) sixteen\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}