{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a56ddf7a-dc7a-5bb0-b6d4-f4b77a1d8df8","task_key":"temporal~5fsequences--main--a56ddf7a-dc7a-5bb0-b6d4-f4b77a1d8df8","task_revision_id":"1","upstream_id":"","short_description":"Today, Nancy went to the bookstore. Between what times could they have gone?","config":"temporal_sequences","split":"main","body":"{\"input\":\"Today, Nancy went to the bookstore. Between what times could they have gone?\\nWe know that:\\nNancy woke up at 8am.\\nJennifer saw Nancy buying cookies at a bakery from 8am to 9am.\\nWilliam saw Nancy taking photos near the Eiffel Tower from 9am to 12pm.\\nMary saw Nancy playing tennis at the tennis court from 12pm to 2pm.\\nThe bookstore was closed after 9pm.\\nBetween what times could Nancy have gone to the bookstore?\\nOptions:\\n(A) 12pm to 2pm\\n(B) 8am to 9am\\n(C) 2pm to 9pm\\n(D) 9am to 12pm\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}