{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6e8ad64b-d279-59a1-86c6-6ac59ba84ea6","task_key":"bbeh~5ftime~5farithmetic--full--6e8ad64b-d279-59a1-86c6-6ac59ba84ea6","task_revision_id":"1","upstream_id":"","short_description":"Let the answer to Question1 be X, Y, Z.","config":"bbeh_time_arithmetic","split":"full","body":"{\"input\":\"Let the answer to Question1 be X, Y, Z.\\n\\nQuestion1: Bob write 2 letters in 1 hour, 37 minutes, and 16 seconds. What is the time taken by them to write 18 letters, at the same pace. Your answer should be the hours, minutes, and seconds, respectively, without the units (for example, 1, 58, 10).\\n\\nLet X' = X - 3, Y' = Y - 18, and Z' = Z - 13.\\n\\nLet the answer to Question2 be P, R, Q.\\n\\nQuestion2: Jimmys's homework time began at X':Y':Z' AM and wrapped up at 04:43:11 PM. How much time passed during his work session? If the end time is before the start time, assume it to be on the next day. Your answer should be the number of hours, minutes, and seconds, respectively, without the units (for example, 13, 6, 47).\\n\\nLet P' = P + 1991, R' = R - 25, and Q' = Q + 32.\\n\\nUse these values to answer Question3.\\n\\nQuestion3: If the date for today in yyyy-mm-dd format is P'-R'-Q', what was the date 5 days ago in the same format? If the question cannot be answered, reply \\\"unanswerable\\\". Use the date format in the question to answer the question.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}