{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f4f395e6-9ecc-5379-a0e0-66b3502aff41","task_key":"default--validation--25356","task_revision_id":"1","upstream_id":"25356","short_description":"HellaSwag validation 25356","config":"default","split":"validation","body":"{\"activity_label\":\"Education and Communications\",\"ctx\":\"[header] How to call latvia [title] Dial your country's exit code. [step] The exit code differs from country to country, so look your code up online if you're unsure what it is. You need to dial this code in order to begin an international call.\",\"endings\":\"['The most common exit codes are 011 and 00. [substeps] The exit code for the u.s.', \\\"[substeps] Some countries will not provide exit codes. If you'll be calling latvia international, they can let you know to call all of the countries and find the area code.\\\", '[substeps] For example, italy has the exit code gb 0600. Australia does not have a country code, but you can look it up in any language.', \\\"[substeps] If you're calling from a landline, dial sankme. If you are calling from an actual landline phone, dial efttpnet.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}