{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0adb2a44-63c8-5a1d-85d2-f71d0bba33cf","task_key":"temporal~5fsequences--main--0adb2a44-63c8-5a1d-85d2-f71d0bba33cf","task_revision_id":"1","upstream_id":"","short_description":"Today, Hannah went to the beach. Between what times could they have gone?","config":"temporal_sequences","split":"main","body":"{\"input\":\"Today, Hannah went to the beach. Between what times could they have gone?\\nWe know that:\\nHannah woke up at 5am.\\nRichard saw Hannah sitting on a rooftop from 5am to 7am.\\nAshley saw Hannah getting a coffee at the cafe from 9am to 7pm.\\nJennifer saw Hannah taking photos near the Leaning Tower of Pisa from 7pm to 8pm.\\nThe beach was closed after 8pm.\\nBetween what times could Hannah have gone to the beach?\\nOptions:\\n(A) 7pm to 8pm\\n(B) 7am to 9am\\n(C) 9am to 7pm\\n(D) 5am to 7am\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}