{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c0faf2d5-4f2e-5384-9d98-cee08fc0bf77","task_key":"bbeh~5fhyperbaton--full--c0faf2d5-4f2e-5384-9d98-cee08fc0bf77","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) massive dog (2) walking bird (3) huge brand-new camera (4) circular earring (5) terrible bag (6) normal-size new car key (7) paper ring (8) silly sunglasses (9) new bird (10) large drill (11) drinking candle (12) russian fire extinguisher (13) iron hat (14) silly scissors (15) new watch (16) circular stapler (17) black wool rectangular house (18) ethiopian book (19) german house (20) rectangular brush (21) huge old-fashioned hammer (22) pyramidal american bowl (23) leather wallet (24) old hammer (25) typing cloth drill (26) eating ball (27) magenta ridiculous apple (28) normal-size new piano (29) pyramidal bag (30) wood sunglasses (31) snorkeling wood brush (32) snorkeling leather wallet (33) indian pencil (34) hiking stapler (35) terrible clock (36) prismlike vietnamese hat (37) cardboard clock (38) fiberglass candle (39) large car key (40) old bed (41) archaic plier (42) exercise scissors (43) whittling car key (44) american bottle (45) terrible bottle (46) mysterious fire extinguisher (47) cloth car key (48) circular turkish ring (49) square table (50) large fork (51) square drill (52) driving shoe (53) violet nice sofa (54) walking rubber necklace (55) wonderful wrench (56) black lovely saw (57) brand-new necklace (58) driving ruler (59) pyramidal american box (60) pyramidal german ring (61) pink saw (62) awful smoking vietnamese drill (63) spherical german fire extinguisher (64) vietnamese marker (65) circular car key\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) old-fashioned large smoking fiberglass pyramidal iranian box\\n(B) big smoking wood triangular indian nice yellow brush\\n(C) blue silly driving iron pyramidal ethiopian car key\\n(D) midsize old exercise fiberglass prismlike american umbrella\\n(E) awful snorkeling rubber prismlike vietnamese huge new necklace\\n(F) crimson repulsive eating glass rectangular bangladeshi old-fashioned candle\\n(G) archaic enormous driving paper square filipino good marker\\n(H) medium-size old-fashioned hiking wool pyramidal german obnoxious marker\\n(I) new big exercise steel square pakistani awful violet scissors\\n(J) tiny archaic driving steel spherical ethiopian mysterious crimson calculator\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}