{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"051c9507-b11d-5a4f-bd59-a0ec31de1c55","task_key":"bbeh~5ftime~5farithmetic--full--051c9507-b11d-5a4f-bd59-a0ec31de1c55","task_revision_id":"1","upstream_id":"","short_description":"Let the answer to Question1 be X/Y/Z.","config":"bbeh_time_arithmetic","split":"full","body":"{\"input\":\"Let the answer to Question1 be X/Y/Z.\\n\\nQuestion1: The match was moved back early by 14 weeks. If the original date for the match was Jun 30, 1999, when will the match take place? Your answer should be the date in the mm/dd/yyyy format (for example, 06/23/2021).\\n\\nLet X' = X + 32.\\n\\nLet the answer to Question2 be P.\\n\\nQuestion2: If today is Thursday, what day of the week was it X' days before now? Your answer should be the day of the week (for example, Sunday).\\n\\nLet Y' = Y - 13 and let P' be the day of the week 60 days after P.\\n\\nLet the answer to Question3 be Q.\\n\\nQuestion3: If it is 03:22 PM on P' , what is the day Y' hours from now? If the information provided is not enough to calculate the answer, or if there is more than one possible answer, reply with unanswerable.\\n\\nLet Z' = Z - 1984 and let Q' be the day of the week 58 days after Q.\\n\\nUse these values to solve Question4.\\n\\nQuestion4: On a Q' before noon, Fred started studying and studied Z' hours non-stop. What day(s) of the week could it be when Fred finished studying? Your answer should be an ordered list with alphabetically ordered items (for example, [answer1, answer2, ...]).\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}