{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"36f68550-05e9-53c0-9467-cdf7849e5764","task_key":"bbeh~5ftime~5farithmetic--full--36f68550-05e9-53c0-9467-cdf7849e5764","task_revision_id":"1","upstream_id":"","short_description":"Let the answer to Question1 be X/Y/Z.","config":"bbeh_time_arithmetic","split":"full","body":"{\"input\":\"Let the answer to Question1 be X/Y/Z.\\n\\nQuestion1: A construction project was due to be completed on 02-28-2007, but has been expedited and finised early by 2 months. What is the new completion date? Your answer should be the date in the mm/dd/yyyy format (for example, 06/23/2021).\\n\\nLet X' = X + 33.\\n\\nLet the answer to Question2 be P.\\n\\nQuestion2: Today is Wednesday and my parents arrive in X' days. On what day of the week do my parents arrive? Your answer should be the day of the week (for example, Sunday).\\n\\nLet Y' = Y - 6 and let P' be the day of the week 38 days after P.\\n\\nLet the answer to Question3 be Q.\\n\\nQuestion3: On P' morning, I was told that the deadline is in Y' days. On what day of the week is the deadline? Your answer should be the day of the week.\\n\\nLet Z' = Z - 1994 and let Q' be the day of the week 76 days after Q.\\n\\nUse these values to solve Question4.\\n\\nQuestion4: If it is Q', what is the day Z' hours from now? If the information provided is not enough to calculate the answer, or if there is more than one possible answer, reply with unanswerable.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}