{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"28173cc7-8cb5-592f-a87a-cb502effc55f","task_key":"bbeh~5fhyperbaton--full--28173cc7-8cb5-592f-a87a-cb502effc55f","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) large car (2) driving umbrella (3) maroon bed (4) archaic driving shoe (5) old walking fork (6) terrible brush (7) indian canvas (8) yellow enormous bird (9) archaic enormous pyramidal pencil (10) rubber repulsive necklace (11) eating pen (12) pyramidal indonesian wrench (13) yellow midsize plate (14) glass bottle (15) circular chinese ring (16) triangular vase (17) paper awful sunglasses (18) magenta midsize earring (19) archaic drinking fork (20) drinking sofa (21) pyramidal speaker (22) walking book (23) new bottle (24) rubber chair (25) red massive camera (26) iron bicycle (27) old-fashioned chair (28) maroon bicycle (29) pyramidal bed (30) rubber awful trash can (31) exercise pink indian book (32) steel piano (33) huge book (34) rubber ruler (35) rubber apple (36) big toolbox (37) yellow guitar (38) ancient marker (39) hiking speaker (40) ancient camera (41) cyan enormous ruler (42) normal-size shirt (43) pakistani sofa (44) new snorkeling fire extinguisher (45) repulsive trash can (46) green medium-size toolbox (47) archaic ruler (48) typing necklace (49) wood spoon (50) prismlike fork (51) new snorkeling apple (52) exercise key (53) tiny watch (54) rectangular dog (55) rectangular thai wallet\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) awful steel gray large rectangular thai whittling old sunglasses\\n(B) yellow enormous circular russian awful steel fork\\n(C) typing green enormous square bangladeshi wool terrible key\\n(D) ancient snorkeling teal normal-size rectangular japanese saw\\n(E) glass green enormous square nigerian whittling old-fashioned toolbox\\n(F) cardboard good black small circular russian typing speaker\\n(G) walking white huge spherical brazilian nice steel spoon\\n(H) terrible yellow midsize pyramidal nigerian drinking ancient ball\\n(I) old eating pink massive square indonesian glass shoe\\n(J) ancient snorkeling maroon enormous circular indian wonderful iron earring\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}