{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"391673ad-398f-58ce-9607-845ea7c09b59","task_key":"bbeh~5fhyperbaton--full--391673ad-398f-58ce-9607-845ea7c09b59","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) brown medium-size sofa (2) pink massive wrench (3) crimson screwdriver (4) wonderful gray exercise earring (5) new trash can (6) iron driving bag (7) russian mysterious marker (8) cardboard house (9) egyptian little plastic knife (10) leather candle (11) driving calculator (12) wool bird (13) black plier (14) pakistani terrible bottle (15) cardboard eating table (16) rubber exercise watch (17) smoking drill (18) enormous candle (19) typing bowl (20) smoking canvas (21) egyptian nice bowl (22) square hammer (23) nice car (24) iron walking banana (25) brazilian good earring (26) chinese ridiculous lamp (27) midsize calculator (28) archaic stapler (29) nice ruler (30) filipino stapler (31) bangladeshi nice brush (32) square spoon (33) old-fashioned lamp (34) iron pencil (35) eating car (36) green jar (37) archaic fork (38) new bird (39) exercise dog (40) green watch (41) wonderful sofa (42) chinese canvas (43) green umbrella (44) terrible shirt (45) triangular calculator (46) old-fashioned pyramidal necklace (47) good canvas (48) iranian clock (49) red enormous cup (50) teal bicycle (51) spherical brush (52) american ball (53) brand-new circular ball (54) terrible box (55) indian ball (56) old-fashioned guitar (57) old prismlike scissors (58) cloth key (59) spherical guitar (60) violet medium-size shirt (61) nigerian mysterious trash can (62) exercise cup (63) prismlike book (64) whittling lamp (65) cardboard necklace (66) magenta little wrench (67) brown saw (68) japanese awful spoon (69) magenta normal-size wrench (70) turkish flower pot (71) paper drinking sunglasses (72) brazilian bird (73) mexican silly book (74) congolese guitar (75) exercise house (76) nigerian ring (77) paper typing umbrella (78) indonesian repulsive flower pot (79) rectangular ruler (80) medium-size bottle (81) blue key (82) smoking trash can (83) new house (84) massive camera (85) ancient calculator (86) rectangular scissors (87) cardboard dog (88) brand-new prismlike ball (89) brown small piano (90) teal flower pot (91) crimson tiny bowl (92) nigerian nice car (93) crimson hammer (94) nice piano (95) nice pencil (96) leather smoking drill (97) congolese knife (98) ancient table (99) bangladeshi drill (100) japanese ridiculous necklace (101) pakistani toolbox (102) leather dog (103) massive chair (104) crimson clock (105) ridiculous canvas (106) american pen (107) glass hiking spoon (108) huge necklace (109) mysterious bicycle\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) pink medium-size iron eating square new car\\n(B) old pink massive leather eating terrible ethiopian flower pot\\n(C) ridiculous teal enormous leather drinking ancient circular speaker\\n(D) american mysterious white big cardboard hiking calculator\\n(E) pyramidal magenta little paper hiking terrible bangladeshi wrench\\n(F) cyan enormous iron typing old-fashioned prismlike pen\\n(G) chinese terrible brown tiny iron whittling triangular ancient brush\\n(H) triangular brand-new brown little cloth snorkeling silly drill\\n(I) american ridiculous white little cardboard exercise new circular lamp\\n(J) circular old-fashioned gray big steel snorkeling car\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}