{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2fe5e5c6-c69a-5813-8990-8f007fd15f57","task_key":"reasoning~5fabout~5fcolored~5fobjects--main--2fe5e5c6-c69a-5813-8990-8f007fd15f57","task_revision_id":"1","upstream_id":"","short_description":"On the desk, there are three orange puzzles, three orange dog leashes, one…","config":"reasoning_about_colored_objects","split":"main","body":"{\"input\":\"On the desk, there are three orange puzzles, three orange dog leashes, one orange envelope, one silver puzzle, three silver fidget spinners, two green fidget spinners, three silver dog leashes, one red envelope, one red dog leash, one green dog leash, two silver envelopes, one green envelope, two green puzzles, and one red puzzle. If I remove all the silver items from the desk, how many dog leashes remain on it?\\nOptions:\\n(A) zero\\n(B) one\\n(C) two\\n(D) three\\n(E) four\\n(F) five\\n(G) six\\n(H) seven\\n(I) eight\\n(J) nine\\n(K) ten\\n(L) eleven\\n(M) twelve\\n(N) thirteen\\n(O) fourteen\\n(P) fifteen\\n(Q) sixteen\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}