{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4248c111-94f1-5744-915b-485e790dbf4c","task_key":"bbeh~5fhyperbaton--full--4248c111-94f1-5744-915b-485e790dbf4c","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) paper congolese brush (2) new scissors (3) big hat (4) wool mexican bag (5) normal-size aqua bowl (6) brown clock (7) blue toolbox (8) big ball (9) iron ancient circular apple (10) prismlike wonderful box (11) cardboard candle (12) glass drill (13) large black chair (14) triangular stapler (15) filipino hat (16) exercise piano (17) obnoxious hammer (18) ancient piano (19) typing piano (20) filipino necklace (21) circular bird (22) prismlike cup (23) mysterious drill (24) fiberglass ethiopian plier (25) old-fashioned lamp (26) tiny crimson canvas (27) iron saw (28) triangular repulsive box (29) little umbrella (30) enormous canvas (31) russian eating repulsive bottle (32) driving old-fashioned hammer (33) glass egyptian speaker (34) cloth drill (35) gray necklace (36) massive book (37) walking shirt (38) russian book (39) plastic nigerian shoe (40) repulsive car key (41) drinking old chair (42) paper canvas (43) old-fashioned canvas (44) indian box (45) typing brand-new plier (46) driving table (47) leather filipino fire extinguisher (48) lovely candle (49) paper russian fire extinguisher (50) steel marker (51) snorkeling archaic screwdriver (52) square obnoxious pen (53) huge blue knife (54) terrible guitar (55) pink wallet (56) walking wrench (57) obnoxious key (58) ancient clock (59) smoking ancient hammer (60) maroon bag (61) pyramidal car key (62) medium-size speaker (63) prismlike bed\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) leather ethiopian whittling brand-new circular mysterious bottle\\n(B) smoking old-fashioned spherical repulsive huge teal stapler\\n(C) egyptian snorkeling ancient spherical silly aqua normal-size umbrella\\n(D) small gray exercise old-fashioned prismlike lovely german wool bag\\n(E) mexican hiking archaic rectangular obnoxious large teal chair\\n(F) black medium-size snorkeling old triangular good russian glass knife\\n(G) wood indonesian driving ancient rectangular awful enormous blue necklace\\n(H) rubber bangladeshi eating ancient spherical silly white normal-size ball\\n(I) iron mexican drinking old-fashioned triangular awful huge fire extinguisher\\n(J) whittling old square awful teal massive brush\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}