{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9831169a-0e58-5848-8f97-1019eeea1e89","task_key":"bbeh~5fhyperbaton--full--9831169a-0e58-5848-8f97-1019eeea1e89","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) cyan clock (2) normal-size table (3) ancient large ring (4) eating sunglasses (5) whittling knife (6) new ruler (7) big spoon (8) brand-new massive bicycle (9) bangladeshi marker (10) rubber stapler (11) massive sofa (12) snorkeling shirt (13) old sofa (14) brand-new tiny bicycle (15) rectangular sunglasses (16) ancient speaker (17) new massive pencil (18) ancient huge fork (19) old hiking yellow knife (20) chinese canvas (21) cyan canvas (22) square ball (23) brown square watch (24) aqua triangular hammer (25) ancient ball (26) fiberglass exercise car (27) magenta prismlike banana (28) brand-new ring (29) pakistani wonderful ruler (30) indonesian drill (31) ancient book (32) driving stapler (33) hiking hammer (34) filipino umbrella (35) hiking marker (36) pakistani clock (37) normal-size earring (38) violet piano (39) black rectangular sofa (40) big bag (41) paper drinking stapler (42) wool eating bag (43) maroon shoe (44) old dog (45) ancient normal-size ball (46) enormous calculator (47) archaic book (48) old tiny shoe (49) driving lamp (50) indian bowl (51) massive candle (52) normal-size flower pot (53) old-fashioned earring (54) teal umbrella (55) prismlike car key (56) old-fashioned small bird (57) new drill (58) archaic little fire extinguisher (59) terrible bag (60) hiking ball (61) pyramidal candle (62) little paper rectangular car key (63) archaic tiny bird (64) pakistani wonderful chair (65) egyptian repulsive candle (66) iron hiking knife (67) brand-new little pencil (68) typing wrench (69) wood whittling wrench (70) wonderful trash can (71) old-fashioned sofa (72) medium-size dog (73) repulsive pencil (74) red spherical canvas (75) midsize knife (76) nigerian awful spoon\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) ancient massive leather eating aqua circular obnoxious bangladeshi book\\n(B) good rubber walking magenta square big brand-new ball\\n(C) repulsive russian rubber hiking yellow circular speaker\\n(D) old-fashioned small glass walking white rectangular good hat\\n(E) mysterious indian paper drinking magenta spherical enormous marker\\n(F) ancient big wool driving teal pyramidal guitar\\n(G) paper driving black spherical thai ridiculous plier\\n(H) silly bangladeshi cardboard hiking teal pyramidal big brand-new plate\\n(I) indian rubber typing green spherical huge ancient marker\\n(J) large wool exercise red spherical good pakistani camera\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}