{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c97ffc2e-16d7-563d-93cf-53cabda3f9f2","task_key":"bbeh~5fhyperbaton--full--c97ffc2e-16d7-563d-93cf-53cabda3f9f2","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) medium-size aqua wrench (2) normal-size spherical repulsive scissors (3) good bird (4) medium-size violet saw (5) filipino scissors (6) prismlike bicycle (7) whittling rectangular wallet (8) iron saw (9) new book (10) exercise circular bag (11) egyptian wallet (12) little green chair (13) circular banana (14) aqua jar (15) pyramidal banana (16) snorkeling pyramidal bed (17) repulsive pakistani stapler (18) snorkeling spherical wallet (19) cloth old hat (20) magenta brush (21) tiny black plier (22) indigo dog (23) ethiopian umbrella (24) drinking flower pot (25) japanese plate (26) normal-size piano (27) indigo toolbox (28) driving bed (29) normal-size plate (30) wool earring (31) nice wallet (32) prismlike trash can (33) steel brand-new umbrella (34) indigo smoking turkish earring (35) glass knife (36) circular dog (37) eating umbrella (38) wonderful bag (39) green brush (40) driving flower pot (41) repulsive egyptian necklace (42) repulsive bag (43) wool ancient shoe (44) brand-new house (45) yellow bird (46) ancient piano (47) awful sofa (48) exercise camera (49) repulsive ethiopian scissors (50) cloth bag (51) rectangular bowl (52) typing square earring (53) huge car (54) old-fashioned knife (55) vietnamese calculator (56) leather sofa (57) exercise saw (58) pyramidal jar (59) iron archaic ruler (60) prismlike chair (61) eating guitar (62) lovely ethiopian toolbox (63) massive gray car key (64) spherical car (65) cloth jar (66) cloth dog (67) rectangular screwdriver (68) good thai stapler (69) spherical calculator (70) cloth archaic umbrella (71) normal-size blue apple (72) snorkeling necklace (73) big teal wallet (74) massive crimson wallet (75) glass book (76) wood shirt (77) spherical screwdriver (78) cardboard ancient stapler (79) smoking pyramidal wrench (80) violet toolbox (81) walking earring (82) german ruler (83) old table (84) snorkeling circular earring (85) tiny aqua plier (86) ancient spoon (87) big jar (88) aqua candle (89) rubber vase (90) big chair (91) midsize bicycle\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) big yellow drinking prismlike mysterious bangladeshi fiberglass brand-new ball\\n(B) old leather drinking pyramidal lovely congolese teal key\\n(C) large gray whittling spherical silly turkish brand-new banana\\n(D) medium-size brown walking triangular ridiculous russian dog\\n(E) little crimson driving circular ridiculous congolese rubber bowl\\n(F) cloth brand-new hiking spherical obnoxious nigerian yellow saw\\n(G) teal snorkeling triangular awful turkish archaic iron wrench\\n(H) red typing circular good vietnamese paper archaic candle\\n(I) typing pyramidal obnoxious mexican cardboard brand-new bag\\n(J) ancient paper smoking circular repulsive turkish bottle\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}