{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"66903ed2-2a1a-5e3a-a750-083681b9a892","task_key":"temporal~5fsequences--main--66903ed2-2a1a-5e3a-a750-083681b9a892","task_revision_id":"1","upstream_id":"","short_description":"Today, Samantha went to the market. Between what times could they have gone?","config":"temporal_sequences","split":"main","body":"{\"input\":\"Today, Samantha went to the market. Between what times could they have gone?\\nWe know that:\\nSamantha woke up at 5am.\\nSarah saw Samantha working at the office from 5am to 11am.\\nAshley saw Samantha buying a phone at the electronics store from 11am to 1pm.\\nLeslie saw Samantha fixing their computer at the electronic store from 5pm to 8pm.\\nThe market was closed after 8pm.\\nBetween what times could Samantha have gone to the market?\\nOptions:\\n(A) 5am to 11am\\n(B) 5pm to 8pm\\n(C) 11am to 1pm\\n(D) 1pm to 5pm\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}