{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4a00fc0d-7b36-5997-9910-fbfac3a84eec","task_key":"bbeh~5ftime~5farithmetic--full--4a00fc0d-7b36-5997-9910-fbfac3a84eec","task_revision_id":"1","upstream_id":"","short_description":"Let the answer to Question1 be X-Y-Z.","config":"bbeh_time_arithmetic","split":"full","body":"{\"input\":\"Let the answer to Question1 be X-Y-Z.\\n\\nQuestion1: The most important game was on 2012-12-19 (yyyy-mm-dd) and the second most important game was 2 days before it. What was the date of the second most important game? If the question cannot be answered, reply \\\"unanswerable\\\". Use the date format in the question to answer the question.\\n\\nLet X' = X - 2003, Y' = Y + 21, and Z' = Z - 4.\\n\\nUse these values to solve Question2.\\n\\nQuestion2: A plane leaves Location A at 09:00 (24hr) UTC+0000 and arrives in Location B X':Y':Z'(hh:mm:ss) later. What time is it in Location B when the plane lands? Location B is in the timezone PST(-0800)). Your answer should include the day and the time. The day should be the number of days before/after the start day (use same_day if the answer is the same day), and time should be in 24-hour format. Some examples are +2, 21:08:10 or same_day, 07:04.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}