{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"853cfe37-5304-56be-ad53-108c6bb41c37","task_key":"bbeh~5ftime~5farithmetic--full--853cfe37-5304-56be-ad53-108c6bb41c37","task_revision_id":"1","upstream_id":"","short_description":"Let the answer to Question1 be X/Y/Z.","config":"bbeh_time_arithmetic","split":"full","body":"{\"input\":\"Let the answer to Question1 be X/Y/Z.\\n\\nQuestion1: If the current date is 30-11-2009, what is the date 19 years from now? If the question asks for a date, your answer should be the date in the mm/dd/yyyy format (for example, 06/23/2021) and if the question asks for a year, in the yyyy format (for example, 2021) .\\n\\nLet X' = X + 26.\\n\\nLet the answer to Question2 be P.\\n\\nQuestion2: On Thursday morning, I was told that the package was out for delivery X' days ago. On what day of the week was the package out for delivery? Your answer should be the day of the week (for example, Sunday).\\n\\nLet Y' = Y - 18 and let P' be the day of the week 40 days after P.\\n\\nLet the answer to Question3 be Q.\\n\\nQuestion3: On P' morning, I was told that the deadline is in Y' days. On what day of the week is the deadline? Your answer should be the day of the week.\\n\\nLet Z' = Z - 1993 and let Q' be the day of the week 15 days after Q.\\n\\nUse these values to solve Question4.\\n\\nQuestion4: On Q' before noon, I received a text saying my deadline is in exactly Z' hours. On what day(s) of the week could the deadline be? Your answer should be an ordered list with alphabetically ordered items (for example, [answer1, answer2, ...]).\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}