{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cb2b8127-6fef-5d2c-b2cd-933b74b7500b","task_key":"temporal~5fsequences--main--cb2b8127-6fef-5d2c-b2cd-933b74b7500b","task_revision_id":"1","upstream_id":"","short_description":"Today, Thomas went to the market. Between what times could they have gone?","config":"temporal_sequences","split":"main","body":"{\"input\":\"Today, Thomas went to the market. Between what times could they have gone?\\nWe know that:\\nThomas woke up at 6am.\\nAndrew saw Thomas sitting on a rooftop from 6am to 7am.\\nElizabeth saw Thomas taking photos near the Leaning Tower of Pisa from 9am to 10am.\\nSamantha saw Thomas waiting at the airport from 10am to 12pm.\\nMichael saw Thomas buying clothes at the mall from 12pm to 6pm.\\nSarah saw Thomas walking towards the Statue of Liberty from 6pm to 7pm.\\nThe market was closed after 7pm.\\nBetween what times could Thomas have gone to the market?\\nOptions:\\n(A) 12pm to 6pm\\n(B) 7am to 9am\\n(C) 6pm to 7pm\\n(D) 10am to 12pm\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}