{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b431904e-9ecb-584b-93bb-e3cc1ddf682d","task_key":"temporal~5fsequences--main--b431904e-9ecb-584b-93bb-e3cc1ddf682d","task_revision_id":"1","upstream_id":"","short_description":"Today, James went to the swimming pool. Between what times could they have gone?","config":"temporal_sequences","split":"main","body":"{\"input\":\"Today, James went to the swimming pool. Between what times could they have gone?\\nWe know that:\\nJames woke up at 5am.\\nAshley saw James sitting on a rooftop from 7am to 9am.\\nHannah saw James taking photos near the Eiffel Tower from 9am to 7pm.\\nWilliam saw James buying cookies at a bakery from 7pm to 8pm.\\nLinda saw James waiting at the train station from 8pm to 10pm.\\nThe swimming pool was closed after 10pm.\\nBetween what times could James have gone to the swimming pool?\\nOptions:\\n(A) 7pm to 8pm\\n(B) 8pm to 10pm\\n(C) 5am to 7am\\n(D) 7am to 9am\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}