{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"24924c82-e63a-5787-bcf1-c635a8a2d072","task_key":"temporal~5fsequences--main--24924c82-e63a-5787-bcf1-c635a8a2d072","task_revision_id":"1","upstream_id":"","short_description":"Today, Andrew went to the coffee shop. Between what times could they have gone?","config":"temporal_sequences","split":"main","body":"{\"input\":\"Today, Andrew went to the coffee shop. Between what times could they have gone?\\nWe know that:\\nAndrew woke up at 8am.\\nJames saw Andrew buying cookies at a bakery from 8am to 1pm.\\nKimberly saw Andrew working at the office from 2pm to 3pm.\\nSarah saw Andrew fixing their computer at the electronic store from 3pm to 5pm.\\nThe coffee shop was closed after 5pm.\\nBetween what times could Andrew have gone to the coffee shop?\\nOptions:\\n(A) 8am to 1pm\\n(B) 2pm to 3pm\\n(C) 1pm to 2pm\\n(D) 3pm to 5pm\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}