{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6ae0925c-6488-5f49-b8a9-65c2beb62b1a","task_key":"bbeh~5fhyperbaton--full--6ae0925c-6488-5f49-b8a9-65c2beb62b1a","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) red bird (2) brand-new square bag (3) ancient car key (4) whittling bicycle (5) nice cup (6) new medium-size leather pen (7) old-fashioned prismlike watch (8) old-fashioned circular wrench (9) archaic pyramidal vase (10) hiking banana (11) paper egyptian flower pot (12) rectangular box (13) drinking large screwdriver (14) mexican bed (15) steel pakistani wrench (16) old-fashioned pyramidal cup (17) triangular bird (18) ancient triangular banana (19) ancient watch (20) indonesian lamp (21) pink terrible toolbox (22) ethiopian canvas (23) glass candle (24) new prismlike chair (25) rectangular speaker (26) drinking earring (27) plastic clock (28) prismlike whittling indonesian apple (29) silly bicycle (30) old plate (31) old spherical box (32) brand-new house (33) brown repulsive speaker (34) congolese ball (35) exercise enormous earring (36) indian sofa (37) huge piano (38) exercise ring (39) green good piano (40) exercise knife (41) bangladeshi cup (42) hiking bed (43) fiberglass american fire extinguisher (44) glass necklace (45) wool book (46) ancient key (47) triangular spoon (48) leather sunglasses (49) typing shoe (50) nice book (51) thai shoe (52) smoking car (53) red terrible stapler (54) cardboard cup (55) iron iranian chair (56) repulsive knife (57) yellow bottle\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) typing normal-size steel iranian nice crimson piano\\n(B) walking massive paper chinese magenta terrible shoe\\n(C) archaic spherical eating normal-size plastic filipino mysterious pink watch\\n(D) awful exercise medium-size wood mexican square old wallet\\n(E) teal awful drinking massive cardboard brazilian rectangular shirt\\n(F) magenta repulsive snorkeling tiny cloth chinese wrench\\n(G) archaic circular exercise huge wood vietnamese calculator\\n(H) archaic rectangular hiking medium-size paper chinese nice knife\\n(I) spherical smoking big cloth egyptian brown ridiculous plier\\n(J) brown mysterious walking tiny wool congolese circular old-fashioned candle\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}