{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f5baa337-c1bd-514b-b9c8-618a8b5d1e90","task_key":"bbeh~5fhyperbaton--full--f5baa337-c1bd-514b-b9c8-618a8b5d1e90","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) good umbrella (2) new plate (3) enormous dog (4) indigo brand-new key (5) cardboard flower pot (6) awful rubber book (7) repulsive fork (8) terrible bicycle (9) bangladeshi large toolbox (10) violet shoe (11) german massive scissors (12) square table (13) circular snorkeling marker (14) silly rubber bed (15) nigerian small plate (16) huge drill (17) russian small canvas (18) snorkeling marker (19) circular driving pen (20) cardboard fork (21) big hat (22) spherical saw (23) ridiculous wool car (24) green car (25) spherical hiking bowl (26) terrible stapler (27) ridiculous car key (28) square driving trash can (29) iron bird (30) nice apple (31) mysterious sunglasses (32) triangular marker (33) gray wrench (34) thai calculator (35) thai small bird (36) maroon new stapler (37) maroon umbrella (38) german spoon (39) indonesian car (40) pyramidal banana (41) hiking jar (42) triangular driving pen (43) brand-new fork (44) smoking cup (45) old-fashioned plate (46) large camera (47) rubber prismlike old saw (48) ancient speaker (49) good car (50) filipino enormous brush (51) iranian bird (52) teal old ring (53) obnoxious walking teal book (54) yellow old trash can (55) little banana (56) good pen (57) medium-size candle (58) old banana (59) red archaic shoe (60) fiberglass hammer (61) red banana (62) brand-new speaker (63) square vase (64) aqua old brush (65) rectangular driving ball (66) midsize bag (67) triangular eating clock (68) pyramidal ball (69) good wood box (70) drinking trash can (71) aqua ancient house (72) mysterious book (73) cardboard speaker (74) terrible steel toolbox (75) circular driving trash can (76) huge apple (77) silly cardboard apple (78) good cup (79) bangladeshi candle (80) japanese screwdriver (81) large screwdriver (82) obnoxious leather box\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) terrible paper triangular eating crimson new thai wrench\\n(B) terrible wood pyramidal eating brown brand-new midsize chinese hat\\n(C) silly iron square typing indigo new egyptian little box\\n(D) little pakistani circular driving indigo old-fashioned wood nice vase\\n(E) medium-size nigerian rectangular snorkeling maroon old-fashioned drill\\n(F) silly plastic circular exercise maroon archaic tiny car key\\n(G) pakistani small triangular snorkeling gray old wood jar\\n(H) steel pyramidal walking red new little mexican stapler\\n(I) good iron square snorkeling blue brand-new box\\n(J) circular exercise maroon old nigerian huge guitar\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}