{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"03438c4a-1819-5183-998f-4d008927d7b1","task_key":"bbeh~5fhyperbaton--full--03438c4a-1819-5183-998f-4d008927d7b1","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) large ridiculous hammer (2) silly sunglasses (3) medium-size ridiculous spoon (4) eating leather watch (5) massive obnoxious bottle (6) new violet saw (7) egyptian circular banana (8) red earring (9) huge mysterious guitar (10) whittling paper flower pot (11) brazilian triangular house (12) mysterious cup (13) ridiculous brush (14) thai pyramidal candle (15) aqua ring (16) pyramidal flower pot (17) congolese spherical bed (18) good piano (19) snorkeling lamp (20) black knife (21) hiking canvas (22) fiberglass car key (23) midsize canvas (24) old red piano (25) steel stapler (26) exercise key (27) brand-new cup (28) nigerian car key (29) brand-new red drill (30) archaic earring (31) indigo shirt (32) driving ball (33) chinese marker (34) smoking cloth car (35) teal dog (36) ancient speaker (37) bangladeshi pencil (38) snorkeling paper ruler (39) new box (40) brand-new yellow brush (41) spherical apple (42) eating cup (43) whittling glass umbrella (44) driving canvas (45) snorkeling pyramidal ancient key (46) iranian triangular fire extinguisher (47) brand-new indigo chair (48) iranian ring (49) eating fiberglass bed (50) brand-new jar (51) wool nigerian crimson bag (52) lovely guitar (53) walking bird (54) old-fashioned magenta scissors (55) iranian cup (56) terrible umbrella (57) walking shoe (58) mysterious wallet (59) russian banana (60) fiberglass necklace (61) brazilian circular plate (62) spherical screwdriver (63) big good stapler (64) old white piano (65) american pyramidal guitar (66) cardboard banana (67) tiny awful saw (68) old book (69) ethiopian rectangular bowl (70) blue clock (71) chinese key (72) eating iron bird (73) iranian prismlike toolbox (74) magenta calculator (75) medium-size shirt (76) little marker (77) pink speaker (78) obnoxious pen (79) huge terrible vase (80) exercise iron ruler (81) ancient sofa (82) congolese spherical flower pot (83) crimson dog (84) large repulsive bottle (85) old shoe (86) new banana (87) hiking fiberglass apple (88) old white hat (89) awful canvas\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) lovely brazilian spherical ancient aqua wool hiking key\\n(B) iron chinese pyramidal brand-new brown silly midsize stapler\\n(C) turkish spherical ancient white massive repulsive watch\\n(D) indian spherical ancient green lovely big shoe\\n(E) midsize awful filipino prismlike old indigo sunglasses\\n(F) midsize mexican spherical brand-new violet glass exercise brush\\n(G) driving glass indonesian pyramidal ancient brown bicycle\\n(H) walking plastic filipino circular old teal good candle\\n(I) massive good german prismlike new pink rubber drinking bird\\n(J) wonderful large german pyramidal old-fashioned black glass stapler\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}