{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1271acc0-c449-5b16-8978-9f5c7e478594","task_key":"temporal~5fsequences--main--1271acc0-c449-5b16-8978-9f5c7e478594","task_revision_id":"1","upstream_id":"","short_description":"Today, John went to the art studio. Between what times could they have gone?","config":"temporal_sequences","split":"main","body":"{\"input\":\"Today, John went to the art studio. Between what times could they have gone?\\nWe know that:\\nJohn woke up at 6am.\\nThomas saw John reading at the library from 7am to 3pm.\\nLisa saw John taking photos near the Eiffel Tower from 3pm to 9pm.\\nSteven saw John buying cookies at a bakery from 9pm to 10pm.\\nThe art studio was closed after 10pm.\\nBetween what times could John have gone to the art studio?\\nOptions:\\n(A) 7am to 3pm\\n(B) 9pm to 10pm\\n(C) 6am to 7am\\n(D) 3pm to 9pm\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}