{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d0f0fce0-4e70-57f1-8603-4dca71835107","task_key":"bbeh~5fhyperbaton--full--d0f0fce0-4e70-57f1-8603-4dca71835107","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) indian saw (2) awful prismlike vase (3) iron banana (4) tiny ridiculous driving saw (5) bangladeshi snorkeling stapler (6) brazilian eating ring (7) ancient fire extinguisher (8) wool fire extinguisher (9) blue normal-size brush (10) smoking chair (11) crimson marker (12) driving marker (13) cloth scissors (14) aqua enormous toolbox (15) wonderful necklace (16) good guitar (17) iranian pen (18) blue jar (19) lovely lamp (20) circular bird (21) indian snorkeling bicycle (22) violet enormous cup (23) wool jar (24) indian driving wrench (25) midsize camera (26) pink rectangular russian plier (27) steel clock (28) ancient steel fork (29) maroon small box (30) brand-new wool dog (31) lovely candle (32) iranian typing vase (33) huge fork (34) mexican typing fork (35) new wool bed (36) spherical chair (37) old-fashioned iron wallet (38) brazilian shoe (39) old-fashioned iron drill (40) mexican bag (41) silly triangular key (42) archaic drill (43) violet enormous wrench (44) ancient key (45) brand-new vase (46) congolese driving book (47) silly clock (48) triangular fork (49) tiny spoon (50) ancient iron fork (51) exercise camera (52) yellow piano (53) iranian eating screwdriver (54) good car key (55) brand-new paper watch (56) wonderful rectangular umbrella (57) exercise flower pot (58) fiberglass cup (59) old-fashioned car key (60) turkish fire extinguisher (61) chinese eating ball (62) brown small bowl (63) japanese eating bicycle (64) medium-size speaker (65) filipino book (66) blue big chair (67) violet midsize ruler (68) ridiculous rectangular table (69) gray large cup (70) nigerian eating jar (71) silly book (72) indonesian screwdriver (73) white enormous candle (74) circular shirt (75) cyan big box (76) vietnamese plier (77) repulsive fork (78) violet enormous trash can (79) maroon brush (80) ethiopian shoe (81) pakistani trash can\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) old-fashioned leather obnoxious prismlike german smoking tiny sofa\\n(B) ancient plastic mysterious prismlike mexican typing canvas\\n(C) rubber archaic nice prismlike pakistani drinking hammer\\n(D) cloth awful pyramidal american snorkeling medium-size brown ring\\n(E) ancient paper silly pyramidal iranian smoking huge crimson sunglasses\\n(F) crimson huge mysterious triangular turkish hiking archaic table\\n(G) cardboard new good spherical russian whittling big banana\\n(H) cyan little nice rectangular pakistani whittling dog\\n(I) pink medium-size nice circular filipino eating ancient steel ball\\n(J) brown enormous terrible square iranian smoking plastic plate\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}