{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c05da7f1-4732-5d1c-b327-c1aedb6c409d","task_key":"bbeh~5fhyperbaton--full--c05da7f1-4732-5d1c-b327-c1aedb6c409d","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) eating camera (2) iranian wallet (3) indonesian screwdriver (4) ancient leather earring (5) lovely car (6) old bowl (7) big egyptian table (8) spherical terrible table (9) hiking white jar (10) driving teal book (11) new cloth wrench (12) ancient rubber key (13) iron plier (14) steel bowl (15) pyramidal piano (16) midsize congolese drill (17) walking black camera (18) wonderful calculator (19) cardboard camera (20) driving brown umbrella (21) walking jar (22) normal-size russian candle (23) snorkeling magenta screwdriver (24) new calculator (25) typing candle (26) ridiculous wrench (27) pyramidal lovely ring (28) steel chair (29) square marker (30) hiking black toolbox (31) brand-new shoe (32) circular ridiculous marker (33) hiking indigo watch (34) normal-size spoon (35) rectangular watch (36) big nigerian calculator (37) pakistani wrench (38) cloth earring (39) circular lovely piano (40) archaic wrench (41) eating aqua sunglasses (42) old brush (43) whittling box (44) paper bag (45) nice pencil (46) triangular repulsive plate (47) blue apple (48) driving umbrella (49) prismlike repulsive necklace (50) wool car key (51) huge cardboard circular knife (52) wonderful watch (53) awful bag (54) awful jar (55) big brazilian saw (56) rectangular ball (57) enormous filipino ball (58) brand-new bed (59) old-fashioned vase (60) enormous filipino sofa (61) wonderful toolbox (62) snorkeling cyan car key (63) pyramidal jar (64) ancient plastic saw (65) spherical obnoxious hammer (66) normal-size book (67) eating plier (68) german chair (69) huge chinese bird (70) new steel wallet (71) old paper fork (72) silly marker (73) wool house (74) spherical hammer (75) snorkeling blue sofa (76) drinking violet guitar (77) new steel ruler (78) spherical banana (79) snorkeling bowl (80) mexican new lovely table\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) typing red old iron prismlike mysterious vietnamese enormous plier\\n(B) eating red ancient wool rectangular ridiculous cup\\n(C) small filipino archaic wool triangular good brown pencil\\n(D) massive turkish old-fashioned fiberglass circular wonderful whittling aqua earring\\n(E) massive mexican archaic leather rectangular terrible aqua walking wallet\\n(F) black eating archaic iron prismlike terrible nigerian lamp\\n(G) maroon walking old-fashioned leather square obnoxious turkish normal-size sunglasses\\n(H) japanese new wool rectangular mysterious crimson hiking camera\\n(I) cyan exercise ancient glass spherical terrible wrench\\n(J) normal-size chinese archaic wool triangular mysterious ball\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}