{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"acfa71fd-c27d-5984-b8ef-de7262098697","task_key":"bbeh~5fhyperbaton--full--acfa71fd-c27d-5984-b8ef-de7262098697","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) brand-new screwdriver (2) glass pencil (3) wool old scissors (4) little clock (5) triangular little speaker (6) mysterious magenta jar (7) old screwdriver (8) indian flower pot (9) vietnamese camera (10) pyramidal plate (11) wood pencil (12) rubber old flower pot (13) triangular ball (14) triangular bag (15) ancient rectangular magenta speaker (16) massive guitar (17) gray cup (18) spherical enormous canvas (19) paper ancient sofa (20) pink scissors (21) drinking guitar (22) leather lamp (23) good apple (24) triangular sunglasses (25) indigo jar (26) ridiculous spoon (27) mysterious trash can (28) cloth sofa (29) ethiopian driving fork (30) triangular normal-size drill (31) cloth archaic fire extinguisher (32) new cup (33) square large book (34) paper jar (35) pyramidal big flower pot (36) wood tiny good plate (37) japanese hiking jar (38) wood jar (39) brazilian hiking car key (40) typing camera (41) good hat (42) circular key (43) fiberglass house (44) terrible shoe (45) walking sunglasses (46) obnoxious magenta wallet (47) repulsive crimson brush (48) repulsive pink flower pot (49) fiberglass ancient bird (50) cloth drill (51) brown hat (52) filipino snorkeling ruler (53) pink screwdriver (54) repulsive scissors (55) japanese pencil (56) american calculator (57) indigo candle (58) midsize marker (59) spherical big camera (60) mexican fork (61) triangular enormous pencil\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) thai spherical little nice maroon new fiberglass canvas\\n(B) leather old rectangular massive ridiculous pink trash can\\n(C) snorkeling pakistani circular small wonderful black ruler\\n(D) circular massive ridiculous white thai exercise bowl\\n(E) paper new prismlike tiny good violet hiking knife\\n(F) exercise indonesian rectangular huge obnoxious pink new glass pencil\\n(G) triangular small nice aqua hiking chinese ring\\n(H) wood brand-new prismlike massive terrible crimson driving indian stapler\\n(I) hiking square massive wonderful magenta old-fashioned fiberglass shoe\\n(J) filipino typing triangular big terrible maroon new plastic ball\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}