{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2216148e-d0c3-5b98-bf7d-82c84263e7e5","task_key":"bbeh~5fhyperbaton--full--2216148e-d0c3-5b98-bf7d-82c84263e7e5","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) big clock (2) plastic stapler (3) square key (4) big house (5) black large jar (6) steel bird (7) spherical camera (8) old key (9) paper box (10) black guitar (11) repulsive glass guitar (12) driving earring (13) midsize camera (14) pyramidal drill (15) pink pencil (16) blue tiny book (17) nigerian old-fashioned stapler (18) walking circular bottle (19) whittling trash can (20) pink large plate (21) lovely paper umbrella (22) walking prismlike toolbox (23) mexican apple (24) old flower pot (25) gray midsize canvas (26) exercise plate (27) black enormous guitar (28) rectangular repulsive midsize plate (29) eating sunglasses (30) american lamp (31) lovely glass apple (32) wood vase (33) large sofa (34) maroon scissors (35) wool hat (36) mysterious glass dog (37) rectangular candle (38) violet medium-size pencil (39) yellow pencil (40) mysterious dog (41) gray large wrench (42) yellow ring (43) iranian old banana (44) thai old-fashioned bed (45) blue toolbox (46) repulsive wool vase (47) large bag (48) lovely umbrella (49) walking square stapler (50) vietnamese jar (51) smoking iron crimson ruler (52) pink bird (53) gray hat (54) green medium-size sunglasses (55) filipino old-fashioned bag (56) huge ruler (57) lovely wood vase (58) maroon stapler (59) square watch (60) brown big wrench (61) new ball (62) nigerian archaic flower pot\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) silly steel black little american archaic table\\n(B) filipino wonderful leather aqua big circular eating shirt\\n(C) driving spherical silly iron cyan large archaic hammer\\n(D) smoking square mysterious plastic red big fire extinguisher\\n(E) smoking spherical wonderful leather gray massive bangladeshi brand-new piano\\n(F) pyramidal repulsive glass teal small bangladeshi old drill\\n(G) exercise square nice wool indigo small old pakistani banana\\n(H) prismlike terrible wool aqua little ancient japanese book\\n(I) archaic russian silly cloth black huge plate\\n(J) old repulsive fiberglass brown enormous rectangular whittling clock\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}