{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e328393a-ac6d-5edd-b60b-5b37516a147c","task_key":"bbeh~5fhyperbaton--full--e328393a-ac6d-5edd-b60b-5b37516a147c","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) snorkeling guitar (2) iron typing candle (3) hiking key (4) ancient square candle (5) typing flower pot (6) brand-new circular fire extinguisher (7) black repulsive bird (8) drinking bed (9) spherical camera (10) turkish shoe (11) white silly calculator (12) wool driving brush (13) plastic speaker (14) russian medium-size brush (15) walking canvas (16) cyan wonderful flower pot (17) prismlike fork (18) brown stapler (19) blue vase (20) wool smoking dog (21) blue obnoxious chair (22) rubber saw (23) wood exercise knife (24) ethiopian saw (25) enormous vase (26) circular guitar (27) brand-new square speaker (28) old pyramidal jar (29) lovely saw (30) iron hiking ball (31) archaic triangular flower pot (32) exercise key (33) cardboard spherical pakistani fire extinguisher (34) smoking shirt (35) old triangular trash can (36) old-fashioned necklace (37) gray terrible trash can (38) old spherical fork (39) new circular camera (40) ridiculous bicycle (41) medium-size stapler (42) big camera (43) chinese massive bicycle (44) obnoxious banana (45) teal chair (46) whittling umbrella (47) cardboard sunglasses (48) old spherical bed (49) rectangular trash can (50) typing shirt (51) obnoxious bed (52) new brush (53) old-fashioned triangular shoe (54) rectangular bed (55) old-fashioned rectangular key (56) cyan terrible watch (57) glass shirt (58) german huge ring (59) ancient rectangular scissors (60) smoking drill (61) pyramidal stapler (62) rectangular car key (63) circular watch (64) egyptian flower pot (65) huge saw (66) small plier (67) yellow clock (68) indigo mysterious umbrella (69) spherical necklace (70) egyptian fire extinguisher (71) archaic wrench (72) glass cup (73) good plier (74) rectangular knife (75) tiny brush (76) maroon house (77) congolese guitar (78) triangular ring (79) egyptian huge bottle (80) indigo chair (81) smoking old massive car key (82) enormous bottle (83) leather cup (84) ethiopian bag (85) huge bird (86) steel smoking sofa (87) smoking shoe (88) russian midsize wrench (89) gray key (90) old-fashioned car key (91) mysterious bicycle (92) russian hat (93) american ball (94) new wrench (95) pyramidal earring (96) iron eating bicycle (97) bangladeshi car key (98) plastic calculator (99) new spherical bag (100) blue banana\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) yellow obnoxious brand-new circular ethiopian huge banana\\n(B) ridiculous indigo new circular american big walking plastic fire extinguisher\\n(C) awful ancient prismlike russian big typing wood shirt\\n(D) green silly archaic circular chinese normal-size drinking paper pencil\\n(E) smoking brand-new pyramidal mexican normal-size violet nice plier\\n(F) archaic circular indian large brown awful camera\\n(G) glass driving old triangular american small bowl\\n(H) iron walking old pyramidal german midsize obnoxious white bed\\n(I) driving ancient spherical vietnamese midsize wonderful violet sunglasses\\n(J) gray wonderful old square vietnamese large smoking pencil\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}