{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"95678c3c-81c8-52a7-99e6-fb9e5d237800","task_key":"temporal~5fsequences--main--95678c3c-81c8-52a7-99e6-fb9e5d237800","task_revision_id":"1","upstream_id":"","short_description":"Today, William went to the coffee shop. Between what times could they have gone?","config":"temporal_sequences","split":"main","body":"{\"input\":\"Today, William went to the coffee shop. Between what times could they have gone?\\nWe know that:\\nWilliam woke up at 9am.\\nLinda saw William playing tennis at the tennis court from 9am to 11am.\\nEmily saw William attending class at the school from 11am to 1pm.\\nNancy saw William walking in the garden from 6pm to 7pm.\\nThe coffee shop was closed after 7pm.\\nBetween what times could William have gone to the coffee shop?\\nOptions:\\n(A) 6pm to 7pm\\n(B) 9am to 11am\\n(C) 11am to 1pm\\n(D) 1pm to 6pm\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}