{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f6a2e628-d19d-55e2-8263-1bd402e64bb3","task_key":"temporal~5fsequences--main--f6a2e628-d19d-55e2-8263-1bd402e64bb3","task_revision_id":"1","upstream_id":"","short_description":"Today, Mary went to the dance studio. Between what times could they have gone?","config":"temporal_sequences","split":"main","body":"{\"input\":\"Today, Mary went to the dance studio. Between what times could they have gone?\\nWe know that:\\nMary woke up at 9am.\\nSarah saw Mary getting a coffee at the cafe from 9am to 10am.\\nMary saw Mary walking in the garden from 12pm to 1pm.\\nTiffany saw Mary driving to the water park from 1pm to 10pm.\\nThe dance studio was closed after 10pm.\\nBetween what times could Mary have gone to the dance studio?\\nOptions:\\n(A) 12pm to 1pm\\n(B) 9am to 10am\\n(C) 1pm to 10pm\\n(D) 10am to 12pm\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}