{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8fa53060-ba21-5076-bd9d-18b25514f06c","task_key":"bbeh~5fhyperbaton--full--8fa53060-ba21-5076-bd9d-18b25514f06c","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) terrible apple (2) awful necklace (3) wood pencil (4) square table (5) massive turkish trash can (6) huge indian shirt (7) fiberglass scissors (8) walking pyramidal canvas (9) paper old-fashioned wrench (10) smoking vase (11) typing circular ball (12) red terrible pen (13) huge box (14) yellow toolbox (15) plastic drill (16) normal-size bowl (17) repulsive book (18) circular sunglasses (19) bangladeshi bed (20) midsize brazilian knife (21) crimson plier (22) white repulsive jar (23) eating watch (24) paper umbrella (25) plastic new marker (26) cardboard ancient toolbox (27) large ethiopian house (28) triangular plier (29) large thai speaker (30) green ring (31) iron screwdriver (32) indigo wrench (33) drinking pyramidal apple (34) silly bottle (35) hiking spherical camera (36) large indonesian candle (37) iron calculator (38) typing triangular flower pot (39) steel shirt (40) brand-new house (41) black car (42) prismlike midsize old-fashioned candle (43) snorkeling cup (44) eating egyptian plastic earring (45) normal-size box (46) american fire extinguisher (47) new bird (48) nice bicycle (49) triangular chair (50) tiny pakistani hammer (51) old car key (52) big congolese sunglasses (53) normal-size mexican piano (54) pakistani trash can (55) german ring (56) smoking plier (57) whittling knife (58) ethiopian clock (59) blue obnoxious toolbox (60) plastic bicycle (61) plastic vase (62) eating circular flower pot (63) old box (64) steel ancient vase (65) walking fork (66) medium-size indonesian table (67) ancient bowl (68) large bowl (69) normal-size bicycle (70) circular plate (71) red bed (72) huge ruler (73) drinking circular spoon (74) japanese spoon (75) silly plate (76) exercise sunglasses (77) mysterious canvas (78) steel stapler (79) brown silly sofa (80) black shirt (81) leather sunglasses (82) hiking earring (83) spherical jar (84) typing bottle (85) massive pencil (86) tiny chinese wallet (87) ethiopian wallet (88) huge pencil (89) ethiopian earring (90) archaic jar (91) steel old-fashioned fork (92) filipino knife (93) fiberglass canvas (94) glass camera (95) archaic stapler (96) huge indian hammer (97) rubber new bird\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) eating rectangular little congolese paper old-fashioned vase\\n(B) yellow awful huge thai wood ancient cup\\n(C) gray large bangladeshi wood new triangular driving house\\n(D) midsize filipino steel old magenta wonderful piano\\n(E) blue repulsive little turkish cardboard brand-new rectangular shoe\\n(F) wonderful aqua large mexican wool archaic rectangular exercise bowl\\n(G) wonderful huge filipino paper old-fashioned spherical exercise bowl\\n(H) mysterious red large bangladeshi cardboard ancient trash can\\n(I) brown terrible little russian glass old-fashioned square driving stapler\\n(J) prismlike normal-size egyptian wood old white nice toolbox\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}