{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d25f73fb-fd65-5ba8-9589-e14f0b58fd94","task_key":"bbeh~5fhyperbaton--full--d25f73fb-fd65-5ba8-9589-e14f0b58fd94","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) ancient fiberglass plier (2) iranian jar (3) vietnamese spherical toolbox (4) ancient screwdriver (5) fiberglass vase (6) pyramidal key (7) wonderful apple (8) magenta bowl (9) ancient cardboard pen (10) driving bed (11) teal typing flower pot (12) triangular saw (13) enormous mysterious speaker (14) brand-new wool sunglasses (15) chinese toolbox (16) lovely necklace (17) vietnamese necklace (18) steel bed (19) repulsive candle (20) triangular lamp (21) archaic fiberglass car key (22) new iron piano (23) eating lamp (24) mysterious knife (25) big plate (26) glass house (27) cloth pen (28) white marker (29) old-fashioned toolbox (30) hiking brand-new lovely bed (31) archaic paper bed (32) smoking scissors (33) driving flower pot (34) white plate (35) terrible table (36) huge stapler (37) white dog (38) spherical lamp (39) spherical fire extinguisher (40) brown wallet (41) eating wallet (42) wool camera (43) tiny pen (44) pyramidal plier (45) american circular bag (46) brown chair (47) black ruler (48) filipino sunglasses (49) repulsive brush (50) archaic steel watch (51) cyan bowl (52) old-fashioned ring (53) nice wallet (54) mysterious brush (55) old-fashioned hammer (56) congolese scissors (57) brand-new fiberglass sofa (58) small silly bowl (59) large terrible vase (60) exercise shoe (61) wool sunglasses (62) driving watch (63) gray walking shirt (64) ancient fire extinguisher (65) archaic spoon (66) massive terrible bag (67) white knife (68) yellow bed (69) congolese rectangular knife (70) old-fashioned sofa (71) nice hat (72) brazilian pyramidal brush (73) ancient cup (74) black cloth normal-size dog (75) circular bicycle (76) tiny box (77) huge wonderful ball (78) whittling marker (79) blue lamp (80) rectangular screwdriver (81) vietnamese triangular marker (82) archaic marker (83) indian canvas (84) pyramidal clock (85) smoking drill (86) brown hiking plate (87) enormous clock (88) archaic key (89) rubber flower pot (90) leather pen (91) gray apple (92) ancient steel drill (93) new iron apple (94) cloth bottle (95) japanese book (96) russian wallet (97) old wood vase (98) walking chair (99) cloth calculator (100) prismlike wrench (101) triangular guitar (102) brand-new fiberglass table (103) square canvas (104) teal flower pot (105) old-fashioned leather hammer (106) violet drinking sunglasses (107) rectangular flower pot (108) blue eating ball (109) old-fashioned canvas (110) leather clock (111) exercise trash can (112) massive pen (113) silly fire extinguisher\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) mexican rectangular old leather midsize awful spoon\\n(B) blue exercise brand-new paper normal-size ridiculous square earring\\n(C) smoking old wood large mysterious square german lamp\\n(D) old rubber normal-size mysterious pyramidal vietnamese trash can\\n(E) circular brand-new leather large ridiculous exercise teal toolbox\\n(F) crimson drinking old-fashioned leather tiny lovely chinese square car\\n(G) archaic plastic large mysterious german rectangular brush\\n(H) japanese square old wool medium-size nice hiking pen\\n(I) snorkeling archaic wood tiny lovely thai circular bottle\\n(J) gray whittling old-fashioned wool tiny mysterious bottle\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}