{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3b6803c7-c474-5b56-8be8-af88738284f5","task_key":"temporal~5fsequences--main--3b6803c7-c474-5b56-8be8-af88738284f5","task_revision_id":"1","upstream_id":"","short_description":"Today, Susan went to the art studio. Between what times could they have gone?","config":"temporal_sequences","split":"main","body":"{\"input\":\"Today, Susan went to the art studio. Between what times could they have gone?\\nWe know that:\\nSusan woke up at 5am.\\nKimberly saw Susan waiting at the train station from 5am to 6am.\\nLisa saw Susan working out at the gym from 6am to 11am.\\nBetty saw Susan reading at the library from 11am to 12pm.\\nThe art studio was closed after 1pm.\\nBetween what times could Susan have gone to the art studio?\\nOptions:\\n(A) 12pm to 1pm\\n(B) 11am to 12pm\\n(C) 6am to 11am\\n(D) 5am to 6am\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}