{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"aaca319f-cc63-5af4-a864-6d9288701d3c","task_key":"bbeh~5fhyperbaton--full--aaca319f-cc63-5af4-a864-6d9288701d3c","task_revision_id":"1","upstream_id":"","short_description":"In a variant of English, we are given that the following sentences have correct…","config":"bbeh_hyperbaton","split":"full","body":"{\"input\":\"In a variant of English, we are given that the following sentences have correct adjective order:\\n(1) smoking house (2) large maroon candle (3) cyan calculator (4) snorkeling rectangular plier (5) indonesian new canvas (6) chinese brand-new fire extinguisher (7) rectangular shirt (8) steel brush (9) russian box (10) awful screwdriver (11) prismlike candle (12) iron pencil (13) awful cloth car key (14) driving triangular wallet (15) normal-size black trash can (16) ancient bag (17) archaic banana (18) normal-size earring (19) egyptian archaic flower pot (20) lovely necklace (21) circular apple (22) congolese brand-new house (23) awful speaker (24) filipino ancient book (25) archaic exercise aqua toolbox (26) repulsive iron scissors (27) driving shoe (28) typing pencil (29) massive gray earring (30) huge pink plier (31) teal key (32) enormous knife (33) vietnamese brand-new saw (34) awful steel vase (35) driving saw (36) eating ring (37) teal sunglasses (38) turkish old key (39) eating car key (40) nigerian pen (41) vietnamese old canvas (42) prismlike fork (43) american ancient pen (44) ridiculous fiberglass bed (45) tiny knife (46) normal-size yellow lamp (47) pyramidal ring (48) driving fire extinguisher (49) german triangular little umbrella (50) glass hat\\n\\nIn this variant of English, which of the following sentences (Options A-J) use the correct adjective order? If none of the sentences (Options A-J) use the correct adjective order, select option K. Select all that apply.\\n(A) fiberglass lovely drinking rectangular tiny pink knife\\n(B) wonderful cloth exercise pyramidal big crimson box\\n(C) hiking triangular little black leather nice house\\n(D) japanese brand-new typing spherical tiny crimson fiberglass awful drill\\n(E) wool whittling square enormous yellow old-fashioned indonesian lamp\\n(F) filipino old-fashioned drinking prismlike little magenta plier\\n(G) iranian ancient snorkeling rectangular tiny teal terrible ball\\n(H) steel terrible eating square huge crimson ancient canvas\\n(I) wood terrible eating triangular big crimson new russian wrench\\n(J) good iron driving square huge brown old-fashioned nigerian apple\\n(K) None of the above\\n\\nProvide your final answer as a concatenation of all the correct choices. For example, if B and C have correct adjective order, then your final answer must be \\\"BC\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}