{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"266797c7-65c6-539f-b011-69d0a3f2b137","task_key":"bbeh~5fhyperbaton--full--266797c7-65c6-539f-b011-69d0a3f2b137","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) fiberglass bicycle (2) gray stapler (3) black good screwdriver (4) leather big cup (5) little stapler (6) enormous book (7) driving dog (8) cardboard ball (9) brown repulsive speaker (10) archaic toolbox (11) plastic brush (12) big bottle (13) maroon banana (14) glass hammer (15) smoking rectangular watch (16) exercise dog (17) maroon bed (18) hiking pencil (19) wood tiny canvas (20) hiking triangular trash can (21) typing shoe (22) filipino vase (23) whittling pyramidal necklace (24) brand-new piano (25) indian wrench (26) glass drill (27) exercise guitar (28) silly vase (29) leather small wallet (30) congolese old-fashioned bag (31) good ruler (32) typing car (33) wool midsize banana (34) exercise pyramidal bird (35) plastic wrench (36) pink sunglasses (37) cardboard umbrella (38) chinese ancient car key (39) snorkeling toolbox (40) iranian necklace (41) green silly bicycle (42) eating spherical necklace (43) eating candle (44) steel enormous hammer (45) japanese ancient scissors (46) mysterious pencil (47) pink earring (48) big necklace (49) cyan obnoxious dog (50) typing house (51) big cup (52) turkish new trash can (53) ancient stapler (54) walking car key (55) white good bird (56) good knife (57) yellow obnoxious pencil (58) cardboard bag (59) normal-size brush (60) green candle (61) congolese knife (62) magenta umbrella (63) wool wrench (64) ancient driving good dog (65) big speaker (66) leather medium-size toolbox (67) good earring (68) exercise square bottle (69) magenta knife (70) leather huge marker (71) circular vase (72) whittling key (73) pakistani circular magenta spoon (74) paper trash can (75) new saw (76) pyramidal bird (77) large bird (78) cyan lovely book (79) rectangular car (80) aqua mysterious apple (81) violet ball (82) exercise piano (83) violet good scissors (84) iranian ancient marker (85) egyptian marker (86) driving jar (87) cloth knife (88) red shoe (89) old-fashioned guitar (90) fiberglass massive knife (91) iranian fire extinguisher (92) steel ball (93) prismlike box (94) nigerian watch (95) repulsive marker (96) blue shirt (97) smoking circular wallet (98) green mysterious ring (99) ridiculous speaker (100) little screwdriver (101) new wallet (102) ethiopian ancient plier (103) triangular fork (104) glass massive sofa (105) filipino archaic guitar (106) obnoxious bowl (107) green lovely bird (108) rubber bird (109) mysterious ruler (110) circular earring\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) new drinking spherical gray obnoxious wood medium-size canvas\\n(B) huge snorkeling spherical cyan awful ancient nigerian plate\\n(C) thai brand-new snorkeling circular aqua good key\\n(D) turkish new driving circular gray lovely midsize glass earring\\n(E) glass little hiking circular teal mysterious old earring\\n(F) mexican old smoking pyramidal yellow nice medium-size hammer\\n(G) congolese brand-new driving spherical magenta silly plastic bird\\n(H) ancient exercise spherical maroon lovely massive paper apple\\n(I) small paper typing rectangular yellow good brand-new nigerian plate\\n(J) chinese old-fashioned typing prismlike brown lovely iron small sofa\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}