{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cd062be2-100f-5e3b-9a64-ba209e5965df","task_key":"bbeh~5fhyperbaton--full--cd062be2-100f-5e3b-9a64-ba209e5965df","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) square ruler (2) vietnamese trash can (3) indonesian bowl (4) black wallet (5) pyramidal vietnamese ball (6) wool bed (7) yellow ridiculous wrench (8) indonesian table (9) circular vietnamese necklace (10) spherical japanese key (11) lovely scissors (12) pyramidal japanese calculator (13) ancient scissors (14) eating old earring (15) black shoe (16) indonesian pencil (17) paper small trash can (18) pyramidal pen (19) ridiculous ruler (20) driving necklace (21) glass medium-size clock (22) midsize plier (23) rectangular guitar (24) medium-size stapler (25) walking scissors (26) typing brand-new ring (27) paper midsize canvas (28) steel trash can (29) new scissors (30) cardboard hammer (31) typing archaic ruler (32) congolese earring (33) prismlike fire extinguisher (34) enormous bed (35) wool midsize pen (36) japanese bed (37) square ethiopian wrench (38) fiberglass sunglasses (39) pink bottle (40) cyan lovely jar (41) brown house (42) plastic bicycle (43) smoking brand-new candle (44) cyan mysterious banana (45) brown bottle (46) prismlike congolese cup (47) nice bed (48) blue jar (49) snorkeling bicycle (50) black guitar (51) paper mysterious rectangular bed (52) leather large knife (53) prismlike piano (54) typing banana (55) driving brand-new pen (56) rectangular indonesian umbrella (57) lovely bottle (58) square indonesian stapler (59) iron big shirt (60) eating new flower pot (61) iron saw (62) brand-new book (63) eating ball (64) mysterious dog (65) silly box (66) enormous dog (67) typing lamp (68) triangular vietnamese plate (69) medium-size red pakistani guitar (70) aqua obnoxious box (71) glass saw (72) triangular ball (73) pyramidal pakistani calculator (74) yellow ridiculous jar (75) repulsive clock (76) russian bag (77) glass little earring (78) wood small canvas (79) good camera (80) red awful bottle (81) steel little car key (82) american sunglasses (83) old-fashioned wallet (84) enormous trash can (85) spherical pakistani wallet (86) plastic little pencil (87) triangular vietnamese marker (88) old-fashioned pencil (89) hiking old-fashioned apple (90) new shirt (91) hiking vase (92) triangular iranian bag (93) cardboard guitar (94) cloth calculator (95) fiberglass medium-size necklace (96) massive drill (97) glass normal-size plate (98) aqua wonderful shirt (99) red candle (100) hiking dog (101) smoking fork (102) hiking plier (103) nigerian bag (104) pyramidal ball (105) terrible banana (106) spherical book\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) cardboard enormous cyan silly spherical indian clock\\n(B) ancient whittling black silly rectangular german massive cardboard toolbox\\n(C) glass small yellow ridiculous spherical brazilian walking new piano\\n(D) rubber midsize pink obnoxious square ethiopian brand-new whittling bag\\n(E) old typing white good square japanese fork\\n(F) little crimson repulsive pyramidal german archaic smoking bicycle\\n(G) snorkeling archaic gray awful circular chinese huge paper apple\\n(H) archaic magenta lovely spherical brazilian medium-size glass table\\n(I) eating old cyan good pyramidal pakistani fork\\n(J) smoking new blue wonderful rectangular thai huge ring\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}