{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e47d4859-0de6-5850-b138-1bc40b02e3fc","task_key":"default--validation--41700","task_revision_id":"1","upstream_id":"41700","short_description":"HellaSwag validation 41700","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to clean a dryer vent on the roof [title] Vacuum on the ground floor. [step] You will need to use a vacuum to clean out the length of the vent and, unless you have access to an industrial vacuum, it is unlikely that you'll be able to do this from the roof. Before climbing up to the roof, get behind the dryer to vacuum out the vent.\",\"endings\":\"['[substeps] Choose a vacuum over an electric one. Electric ones are notorious for damaging metal or mechanical equipment.', '[substeps] There should be clamps that connect the exhaust pipe of the dryer to the vent. Use a screwdriver to release these so that you can vacuum.', \\\"It's likely that you won't be able to reach the vents from the ground floor. [title] Use a water hose if you can't access the vents from the roof.\\\", '[substeps] The exhaust supply tube should be long enough to comfortably reach the vent from the roof. [title] Turn off the dryer and remove the interior plug wire.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}