{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"bebf70c5-cf16-5d72-b2fb-c0153cc11157","task_key":"default--validation--3053","task_revision_id":"1","upstream_id":"3053","short_description":"HellaSwag validation 3053","config":"default","split":"validation","body":"{\"activity_label\":\"Computers and Electronics\",\"ctx\":\"[header] How to buy glasses that fit [title] Have an eye examination. [step] To order glasses that fit from an optometrist, you'll need to have an up-to-date eye examination on file with their office. This usually means an exam that was performed within the last year.\",\"endings\":\"['The optometrist will likely ask you about your prescription. Know when you register your prescription, and what tools or equipment you use to obtain the correct lenses.', \\\"If you haven't had an exam in the last 12 months, you'll need to schedule one. [substeps] Prepare for an eye exam by scheduling it with your eye doctor.\\\", '[substeps] Most optometrists will not take x-rays of sensitive areas, such as the eye canals. They may require one to one imaging or vision examinations to make sure that glasses fit properly.', \\\"Tell them you want to see glasses that get altered over time and that you're looking for glasses that are rigid, while also asking how the glasses fit you. [substeps] These scans can usually be performed at a store that sells prescription glasses.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}