{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e8dbe541-cdf1-54ca-b8fc-07403dcc7f95","task_key":"bbeh~5fhyperbaton--full--e8dbe541-cdf1-54ca-b8fc-07403dcc7f95","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) nigerian wood fork (2) midsize scissors (3) whittling old-fashioned scissors (4) white enormous trash can (5) thai book (6) pyramidal silly camera (7) circular house (8) gray canvas (9) wool sofa (10) pyramidal saw (11) whittling archaic calculator (12) exercise lamp (13) cloth shirt (14) normal-size drill (15) red enormous watch (16) nice sofa (17) violet large sofa (18) black shoe (19) pink shirt (20) walking old-fashioned spoon (21) small calculator (22) midsize stapler (23) glass bag (24) white enormous calculator (25) midsize screwdriver (26) typing new key (27) hiking hammer (28) ethiopian clock (29) drinking shoe (30) mexican cardboard jar (31) repulsive earring (32) american dog (33) square silly sunglasses (34) teal old-fashioned filipino clock (35) indian wool scissors (36) spherical piano (37) triangular wonderful pencil (38) mexican cardboard necklace (39) eating cup (40) congolese wool plate (41) hiking brand-new shoe (42) repulsive stapler (43) midsize ball (44) brand-new bed (45) drinking old-fashioned saw (46) little piano (47) black big banana (48) eating old-fashioned sofa (49) rectangular plate (50) awful hammer (51) little flower pot (52) old apple (53) massive cup (54) drinking new spoon (55) triangular lovely necklace (56) leather sofa (57) good canvas (58) snorkeling brand-new bag (59) spherical car (60) whittling brand-new bird (61) old-fashioned ruler (62) rubber camera (63) large typing wool ruler (64) ancient dog\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) circular terrible whittling archaic ethiopian rubber small key\\n(B) magenta massive exercise archaic pakistani rubber prismlike repulsive speaker\\n(C) driving brand-new japanese rubber prismlike lovely scissors\\n(D) prismlike whittling new japanese rubber midsize teal piano\\n(E) white big whittling old-fashioned american cloth terrible triangular umbrella\\n(F) yellow enormous walking old egyptian iron spherical knife\\n(G) indigo normal-size driving new indian wood camera\\n(H) triangular repulsive smoking old-fashioned pakistani plastic small teal shirt\\n(I) normal-size driving old thai wool wonderful circular spoon\\n(J) crimson medium-size exercise old-fashioned filipino cloth silly umbrella\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}