{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7d74fd0a-0e61-5118-a313-e923308026ab","task_key":"bbeh~5fhyperbaton--full--7d74fd0a-0e61-5118-a313-e923308026ab","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) big smoking banana (2) magenta screwdriver (3) archaic iron sofa (4) mysterious sofa (5) obnoxious ball (6) chinese camera (7) large jar (8) old wood car (9) archaic candle (10) awful scissors (11) old-fashioned apple (12) indigo awful bed (13) aqua nice flower pot (14) white camera (15) spherical mexican car key (16) maroon terrible banana (17) black screwdriver (18) walking hammer (19) pink ridiculous stapler (20) triangular drill (21) indigo obnoxious camera (22) blue ridiculous dog (23) repulsive hat (24) huge hat (25) large snorkeling toolbox (26) pink spoon (27) massive bowl (28) rectangular american wrench (29) brand-new glass cup (30) black plier (31) normal-size german green sunglasses (32) new plier (33) obnoxious canvas (34) brand-new sunglasses (35) old car (36) enormous whittling wrench (37) aqua scissors (38) exercise prismlike lovely watch (39) medium-size snorkeling camera (40) good bowl (41) violet banana (42) smoking calculator (43) steel knife (44) obnoxious brush (45) maroon key (46) spherical screwdriver (47) chinese shirt (48) midsize walking car (49) square russian drill (50) archaic cloth sunglasses (51) triangular american plate (52) midsize snorkeling sunglasses (53) thai saw (54) whittling canvas\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) whittling square turkish magenta silly steel new cup\\n(B) enormous driving square iranian yellow nice old spoon\\n(C) enormous typing triangular brazilian aqua repulsive vase\\n(D) drinking rectangular mexican maroon mysterious old-fashioned iron ball\\n(E) circular american blue repulsive ancient steel sunglasses\\n(F) enormous drinking triangular ethiopian green wonderful glass jar\\n(G) rubber archaic prismlike turkish indigo wonderful drinking midsize saw\\n(H) steel spherical iranian black mysterious exercise large calculator\\n(I) old plastic circular brazilian white wonderful typing midsize apple\\n(J) rubber new prismlike pakistani teal awful typing brush\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}