{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"20719c92-6d15-5687-b7f6-96174dbb26da","task_key":"bbeh~5fhyperbaton--full--20719c92-6d15-5687-b7f6-96174dbb26da","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) circular wrench (2) normal-size banana (3) congolese little fire extinguisher (4) american enormous sofa (5) maroon saw (6) american bottle (7) brazilian little speaker (8) mexican ring (9) crimson good car (10) old-fashioned pencil (11) large hammer (12) nice hat (13) circular leather pencil (14) egyptian medium-size scissors (15) cyan chair (16) ancient hiking house (17) turkish clock (18) yellow sofa (19) violet watch (20) yellow stapler (21) brazilian large knife (22) repulsive stapler (23) blue table (24) triangular paper drill (25) congolese paper brand-new chair (26) new hiking pen (27) bangladeshi speaker (28) pyramidal glass scissors (29) nigerian medium-size piano (30) hiking toolbox (31) square steel bird (32) spherical rubber bowl (33) midsize ruler (34) teal terrible chair (35) old hammer (36) pyramidal jar (37) massive jar (38) old-fashioned calculator (39) mexican huge saw (40) pyramidal wool knife (41) old drinking canvas (42) aqua nice speaker (43) paper watch (44) rectangular drill (45) pyramidal apple (46) little bag (47) ancient drill (48) square knife (49) large pyramidal smoking flower pot (50) brand-new flower pot (51) new whittling calculator (52) cardboard bicycle (53) brown silly bag (54) circular bottle (55) ethiopian marker (56) fiberglass trash can (57) congolese shoe (58) aqua terrible trash can (59) iranian large pencil (60) maroon repulsive bed (61) typing canvas (62) rectangular spoon (63) glass umbrella (64) driving bicycle (65) small apple (66) blue ridiculous toolbox (67) medium-size table (68) rectangular cup (69) old wrench (70) japanese large apple (71) awful wrench (72) steel table (73) pink nice sunglasses (74) leather key (75) tiny brush (76) mysterious brush (77) typing stapler (78) little scissors (79) mexican sunglasses (80) red speaker (81) square iron guitar (82) turkish midsize lamp (83) indian enormous bag (84) ancient whittling hammer (85) magenta obnoxious earring (86) chinese midsize bird (87) smoking dog (88) whittling ruler (89) indonesian medium-size trash can (90) ancient guitar (91) cardboard brush (92) cyan wonderful sofa (93) rectangular paper fork (94) crimson silly plier (95) indonesian dog (96) triangular scissors (97) new wrench (98) old drinking necklace (99) green watch (100) white fork (101) brand-new drinking lamp (102) pyramidal wool candle (103) midsize plate (104) german normal-size table (105) white bowl (106) old-fashioned drinking apple (107) huge saw\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) rectangular cardboard archaic driving good red wrench\\n(B) white awful rectangular cloth brand-new typing massive german fire extinguisher\\n(C) large circular iron ancient snorkeling maroon lovely ruler\\n(D) triangular paper new walking black wonderful cup\\n(E) nigerian massive spherical cardboard old-fashioned walking obnoxious pink fork\\n(F) mexican big prismlike rubber old exercise aqua silly fork\\n(G) japanese tiny spherical cloth ancient hiking piano\\n(H) blue nice square glass old-fashioned walking medium-size calculator\\n(I) terrible brown circular fiberglass old-fashioned driving huge russian car key\\n(J) big triangular cloth ancient whittling awful gray canvas\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}