{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"24c71280-7325-5ce8-ae62-2c37e17eec83","task_key":"temporal~5fsequences--main--24c71280-7325-5ce8-ae62-2c37e17eec83","task_revision_id":"1","upstream_id":"","short_description":"Today, Samantha went to the movies. Between what times could they have gone?","config":"temporal_sequences","split":"main","body":"{\"input\":\"Today, Samantha went to the movies. Between what times could they have gone?\\nWe know that:\\nSamantha woke up at 5am.\\nLisa saw Samantha driving to the water park from 8am to 12pm.\\nMary saw Samantha walking in the garden from 12pm to 4pm.\\nSamantha saw Samantha reading at the library from 4pm to 6pm.\\nThe movies was closed after 6pm.\\nBetween what times could Samantha have gone to the movies?\\nOptions:\\n(A) 8am to 12pm\\n(B) 4pm to 6pm\\n(C) 5am to 8am\\n(D) 12pm to 4pm\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}