{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"17c1133d-89d6-5404-bfad-09c24455334e","task_key":"bbeh~5fhyperbaton--full--17c1133d-89d6-5404-bfad-09c24455334e","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) brand-new umbrella (2) huge car key (3) drinking flower pot (4) blue fire extinguisher (5) ethiopian jar (6) brown table (7) brown drinking sunglasses (8) old-fashioned car (9) obnoxious scissors (10) silly indonesian plate (11) typing necklace (12) hiking trash can (13) awful brazilian camera (14) new rectangular clock (15) bangladeshi key (16) plastic bicycle (17) exercise enormous filipino car (18) large knife (19) old triangular drill (20) mexican fork (21) new fork (22) blue whittling wrench (23) silly shoe (24) enormous sofa (25) normal-size hammer (26) new rectangular book (27) exercise knife (28) crimson plastic ridiculous piano (29) magenta bag (30) new guitar (31) green umbrella (32) prismlike guitar (33) good knife (34) indigo eating fork (35) square pen (36) terrible bottle (37) blue scissors (38) snorkeling scissors (39) crimson snorkeling hammer (40) cardboard chair (41) walking lamp (42) old-fashioned rectangular scissors (43) driving bowl (44) exercise scissors (45) ancient fork (46) obnoxious chair (47) big earring (48) tiny steel banana (49) wonderful congolese dog (50) spherical table (51) massive wool earring (52) pyramidal pen (53) old car\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) prismlike brand-new normal-size wool good vietnamese hiking violet ring\\n(B) walking tiny cloth lovely japanese pyramidal new plate\\n(C) walking midsize leather repulsive pakistani archaic spherical dog\\n(D) green hiking huge plastic terrible american new square toolbox\\n(E) normal-size leather repulsive egyptian pyramidal brand-new shoe\\n(F) circular medium-size cardboard obnoxious iranian typing yellow ball\\n(G) brand-new triangular huge glass obnoxious vietnamese dog\\n(H) aqua driving normal-size steel obnoxious pakistani saw\\n(I) old-fashioned square large steel lovely turkish whittling knife\\n(J) rectangular new massive paper silly american snorkeling saw\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}