{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a4f223e1-cc40-55ed-b64e-b39728ce2971","task_key":"bbeh~5fhyperbaton--full--a4f223e1-cc40-55ed-b64e-b39728ce2971","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) exercise egyptian shoe (2) old piano (3) eating banana (4) lovely huge bicycle (5) lovely candle (6) small apple (7) terrible normal-size necklace (8) wonderful gray driving bicycle (9) paper black shirt (10) snorkeling indian calculator (11) repulsive bag (12) smoking mexican vase (13) snorkeling ring (14) nice enormous pencil (15) triangular banana (16) walking mexican car key (17) silly massive candle (18) lovely normal-size knife (19) exercise jar (20) good screwdriver (21) ridiculous bed (22) midsize iron mexican jar (23) paper blue car (24) driving chinese sofa (25) brand-new calculator (26) square bed (27) brown canvas (28) ethiopian sunglasses (29) terrible guitar (30) small earring (31) indonesian dog (32) awful speaker (33) rectangular shoe (34) big vase (35) iron bowl (36) eating house (37) blue earring (38) black camera (39) huge cup (40) square car key (41) archaic circular speaker (42) ancient hat (43) indonesian lamp (44) exercise brush (45) brand-new triangular fire extinguisher (46) cloth banana (47) normal-size lamp (48) new rectangular cup (49) snorkeling american spoon (50) spherical saw (51) typing shoe (52) whittling book (53) circular bird (54) indonesian key (55) magenta bowl (56) german sofa (57) egyptian ruler (58) teal plate (59) cyan dog (60) snorkeling chinese jar (61) steel black pencil (62) paper banana (63) brand-new jar (64) tiny earring (65) repulsive large wallet (66) paper pink vase (67) wonderful candle (68) old-fashioned necklace (69) magenta house (70) paper violet clock (71) blue umbrella (72) leather pink bottle (73) square guitar (74) ethiopian car key (75) tiny umbrella\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) good large leather red hiking german brand-new key\\n(B) circular glass pink hiking mexican little repulsive saw\\n(C) triangular old-fashioned iron pink eating turkish midsize canvas\\n(D) old cardboard black snorkeling nigerian large terrible fork\\n(E) terrible midsize cloth green walking american old pyramidal box\\n(F) awful tiny glass yellow smoking bangladeshi wrench\\n(G) lovely little plastic red drinking american pyramidal bowl\\n(H) old pyramidal leather violet whittling turkish chair\\n(I) ancient rectangular steel violet eating mexican normal-size wonderful watch\\n(J) medium-size plastic white walking bangladeshi old rectangular hat\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}