{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c31da796-ba7c-5717-a645-7803968c3e22","task_key":"default--validation--22309","task_revision_id":"1","upstream_id":"22309","short_description":"HellaSwag validation 22309","config":"default","split":"validation","body":"{\"activity_label\":\"Education and Communications\",\"ctx\":\"[header] How to call guatemala [title] Dial your country's exit code. [step] Every country has what is called an exit code. It's the number you dial in order to direct a call outside the country.\",\"endings\":\"['You may have to dial this number several times to reach guatemala. You can also use different numbers in different countries to reach guatemala.', \\\"Its final digit (- 123) is 4 digits long, and indicates that you will dial certain numbers outside the country. [substeps] If you're calling the u.s.\\\", 'To make sure the country you\\\\'re calling isn\\\\'t the same as guatemala : [substeps] Replace the 9-digit exit code with a 4-digit area code. ( unless you\\\\'ve previously dialed \\\" 800 \\\").', 'In the united states and canada, the exit code is \\\" 011. \\\" if calling from the us or canada, dial these numbers first.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}