{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ee2f4b85-4a53-54ed-86bf-e89ac63702c9","task_key":"bbeh~5fhyperbaton--full--ee2f4b85-4a53-54ed-86bf-e89ac63702c9","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) red chinese guitar (2) glass scissors (3) repulsive scissors (4) terrible archaic scissors (5) thai apple (6) rubber pencil (7) walking pen (8) typing dog (9) normal-size house (10) ancient sunglasses (11) violet german bag (12) big pyramidal sofa (13) thai flower pot (14) drinking plastic dog (15) wool brush (16) midsize drill (17) little flower pot (18) old smoking nigerian fork (19) black knife (20) black pakistani book (21) whittling cloth screwdriver (22) mysterious ancient brush (23) spherical jar (24) teal plate (25) small earring (26) whittling fiberglass calculator (27) old piano (28) hiking paper spoon (29) wonderful ring (30) enormous rectangular canvas (31) mysterious rubber maroon wallet (32) tiny spoon (33) prismlike pen (34) driving cardboard apple (35) plastic cup (36) smoking apple (37) russian speaker (38) silly ancient earring (39) silly key (40) little prismlike toolbox (41) aqua necklace (42) magenta indonesian sunglasses (43) eating camera (44) indonesian banana (45) ancient trash can (46) awful apple (47) nice sunglasses (48) midsize pyramidal knife (49) mexican bicycle (50) prismlike necklace (51) terrible old-fashioned screwdriver (52) brand-new flower pot (53) old marker (54) steel fork (55) nice necklace (56) archaic key (57) good ball (58) tiny bottle (59) nice dog (60) teal vietnamese marker (61) good old jar (62) american pen (63) enormous pyramidal bottle (64) turkish umbrella (65) indonesian ruler (66) mysterious brand-new house (67) lovely shoe (68) tiny knife (69) obnoxious calculator (70) old-fashioned pencil (71) terrible bicycle (72) awful brush (73) turkish jar (74) pyramidal wallet (75) old table (76) massive stapler (77) little circular vase (78) circular umbrella (79) german bag (80) old-fashioned pen (81) old screwdriver (82) eating banana (83) typing cloth knife (84) small square earring (85) smoking plastic pencil (86) triangular wallet (87) walking wrench (88) enormous banana (89) hiking car key (90) little square bottle (91) obnoxious sofa (92) aqua bed (93) paper speaker (94) square marker (95) snorkeling guitar (96) little calculator (97) hiking piano (98) spherical earring (99) magenta saw (100) cardboard necklace (101) drinking wallet\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) snorkeling cloth red indian tiny prismlike speaker\\n(B) lovely brand-new driving glass gray egyptian vase\\n(C) lovely archaic hiking steel crimson turkish square enormous pencil\\n(D) small whittling fiberglass pink bangladeshi new ridiculous hammer\\n(E) ancient driving leather red indonesian triangular midsize ruler\\n(F) silly new typing fiberglass magenta iranian normal-size rectangular earring\\n(G) silly new whittling fiberglass pink russian normal-size calculator\\n(H) tiny spherical driving iron black bangladeshi ring\\n(I) circular hiking iron gray russian brand-new awful ball\\n(J) enormous spherical walking wood black brazilian old-fashioned silly plier\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}