{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cdc4f9fc-4f12-5780-b468-476f20150a47","task_key":"bbeh~5fhyperbaton--full--cdc4f9fc-4f12-5780-b468-476f20150a47","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) awful archaic shirt (2) square canvas (3) obnoxious bottle (4) indian tiny book (5) walking blue plier (6) snorkeling crimson shoe (7) ridiculous ball (8) old fire extinguisher (9) big sofa (10) german piano (11) square cloth piano (12) new camera (13) large fire extinguisher (14) ethiopian banana (15) enormous shoe (16) japanese apple (17) bangladeshi clock (18) glass fork (19) nice archaic apple (20) brazilian huge bed (21) driving blue calculator (22) terrible ancient stapler (23) obnoxious brand-new bottle (24) snorkeling sunglasses (25) medium-size sofa (26) mexican bird (27) hiking cyan flower pot (28) teal pen (29) spherical brand-new eating clock (30) american large bottle (31) new car key (32) ancient fire extinguisher (33) prismlike wood shoe (34) rectangular steel hammer (35) snorkeling yellow hammer (36) indonesian medium-size key (37) black flower pot (38) iron screwdriver (39) hiking ball (40) massive toolbox (41) spherical wool bottle (42) indonesian marker (43) pyramidal saw (44) vietnamese huge clock (45) brand-new hammer (46) drinking chair (47) little guitar (48) repulsive umbrella (49) american hat (50) good ancient bowl (51) wool toolbox (52) circular scissors (53) walking hammer (54) new shirt (55) prismlike plate (56) obnoxious sofa (57) huge bird (58) terrible brand-new banana (59) iron box (60) triangular leather guitar (61) gray fork (62) old bowl (63) nigerian wrench (64) teal bowl (65) pakistani huge necklace (66) hiking scissors (67) crimson sofa (68) paper car (69) drinking brown drill (70) ethiopian trash can (71) awful ancient shoe (72) american bed (73) prismlike speaker (74) wood lovely teal camera (75) spherical glass clock (76) old-fashioned pen (77) red bottle (78) fiberglass box (79) filipino large sunglasses (80) repulsive bowl (81) triangular leather brush (82) blue cup (83) silly wallet (84) smoking violet watch (85) good jar (86) cloth cup (87) wonderful ring (88) brown knife (89) archaic fork (90) old-fashioned dog (91) archaic bag (92) ethiopian midsize speaker (93) german wallet (94) magenta trash can\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) iranian small silly brand-new walking yellow cloth bed\\n(B) rubber obnoxious brand-new snorkeling magenta indian large speaker\\n(C) tiny bangladeshi terrible brand-new snorkeling magenta glass watch\\n(D) chinese big wonderful new whittling gray cup\\n(E) thai awful old hiking black iron circular camera\\n(F) steel nice old whittling maroon medium-size filipino hat\\n(G) prismlike wool ridiculous ancient driving cyan thai car\\n(H) spherical paper wonderful ancient smoking pink large fork\\n(I) indian medium-size wonderful old snorkeling white wool triangular candle\\n(J) square glass obnoxious archaic drinking crimson ring\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}