{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f4495d8a-a4b3-5ed5-ab74-da2d449a7cf8","task_key":"bbeh~5fhyperbaton--full--f4495d8a-a4b3-5ed5-ab74-da2d449a7cf8","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) enormous crimson car key (2) tiny bicycle (3) huge brown jar (4) midsize white umbrella (5) iranian new candle (6) smoking piano (7) black hammer (8) leather pen (9) cyan table (10) whittling bird (11) turkish marker (12) ridiculous rectangular scissors (13) steel bottle (14) triangular wrench (15) aqua book (16) cloth book (17) lovely prismlike marker (18) chinese archaic dog (19) repulsive cup (20) bangladeshi new house (21) square drill (22) typing enormous prismlike calculator (23) obnoxious pyramidal necklace (24) japanese pencil (25) black plier (26) silly hammer (27) wool whittling apple (28) rectangular plate (29) awful square saw (30) old flower pot (31) tiny vase (32) good table (33) terrible square bird (34) red necklace (35) rubber bottle (36) drinking ring (37) nice triangular shoe (38) typing bicycle (39) hiking shoe (40) plastic walking shoe (41) eating drill (42) awful prismlike car (43) huge pink scissors (44) normal-size magenta watch (45) american apple (46) square candle (47) large sofa (48) triangular speaker (49) obnoxious pyramidal saw (50) medium-size violet shoe (51) spherical sofa (52) ancient clock (53) drinking ball (54) enormous magenta umbrella (55) glass walking piano (56) rubber white repulsive drill (57) whittling wrench (58) japanese brand-new bag (59) typing plier (60) paper typing bag (61) typing umbrella (62) pyramidal trash can (63) archaic shirt (64) small apple (65) obnoxious shirt (66) wool knife (67) old-fashioned shoe (68) obnoxious ruler (69) ancient necklace (70) japanese toolbox (71) massive cyan scissors (72) cloth drill (73) snorkeling ring (74) indonesian shirt (75) nice square bird (76) rubber pen (77) wonderful earring (78) glass typing guitar\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) japanese massive teal awful circular typing steel banana\\n(B) new pakistani tiny crimson lovely pyramidal key\\n(C) steel whittling medium-size magenta lovely circular brazilian umbrella\\n(D) little yellow mysterious pyramidal nigerian old-fashioned flower pot\\n(E) glass driving normal-size black terrible spherical ancient brazilian key\\n(F) german old-fashioned little yellow wonderful square smoking cardboard necklace\\n(G) eating medium-size black silly pyramidal russian old-fashioned spoon\\n(H) old-fashioned small blue awful rectangular exercise rubber screwdriver\\n(I) fiberglass whittling small teal mysterious prismlike chinese archaic knife\\n(J) fiberglass snorkeling tiny maroon repulsive prismlike apple\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}