{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7a3a9e98-8982-5614-9df9-88c6b2ea96fd","task_key":"bbeh~5fhyperbaton--full--7a3a9e98-8982-5614-9df9-88c6b2ea96fd","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) whittling little wood apple (2) chinese crimson stapler (3) new massive canvas (4) brand-new necklace (5) old drill (6) turkish camera (7) ancient big ring (8) lovely pen (9) eating triangular earring (10) typing sofa (11) eating spherical ruler (12) large scissors (13) cardboard pen (14) little pen (15) pyramidal candle (16) normal-size ruler (17) circular jar (18) glass bicycle (19) archaic sunglasses (20) obnoxious apple (21) small bowl (22) black shirt (23) fiberglass canvas (24) crimson house (25) pyramidal key (26) crimson piano (27) old large bicycle (28) leather wrench (29) congolese violet flower pot (30) ancient midsize fire extinguisher (31) brand-new normal-size bowl (32) cloth earring (33) maroon speaker (34) smoking shoe (35) eating banana (36) new large vase (37) prismlike banana (38) triangular house (39) exercise triangular necklace (40) silly plier (41) small hat (42) new massive stapler (43) drinking triangular house (44) triangular canvas (45) mexican green bird (46) brand-new little ruler (47) archaic midsize vase (48) massive shoe (49) square plier (50) smoking square drill (51) nice box (52) rubber guitar (53) square pen (54) steel repulsive bag (55) massive car (56) chinese ring (57) pink sofa (58) ethiopian indigo banana (59) silly piano (60) drinking bottle (61) cardboard house (62) brand-new hammer (63) snorkeling bird (64) cardboard lamp (65) red fire extinguisher (66) good bicycle (67) walking square ruler (68) bangladeshi pink wallet (69) plastic key (70) new large hat (71) red flower pot (72) iron sofa (73) thai magenta ball (74) circular car key (75) crimson candle (76) enormous knife (77) spherical pen (78) spherical brand-new awful candle (79) rectangular hat\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) walking triangular ancient tiny wool terrible sunglasses\\n(B) spherical brand-new normal-size steel wonderful blue filipino sunglasses\\n(C) square old-fashioned little steel wonderful ethiopian indigo vase\\n(D) snorkeling circular archaic little cardboard good egyptian violet flower pot\\n(E) eating pyramidal old small fiberglass mysterious teal russian bowl\\n(F) teal pakistani old-fashioned massive steel mysterious chair\\n(G) drinking rectangular old-fashioned enormous wood silly teal hammer\\n(H) thai magenta brand-new medium-size leather wonderful spoon\\n(I) pakistani aqua old enormous paper wonderful pyramidal typing candle\\n(J) old-fashioned big cloth wonderful vietnamese crimson wallet\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}