{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8b4fa384-113f-5744-9540-46866a85f6e7","task_key":"default--validation--37021","task_revision_id":"1","upstream_id":"37021","short_description":"HellaSwag validation 37021","config":"default","split":"validation","body":"{\"activity_label\":\"Youth\",\"ctx\":\"[header] How to use a blackhead vacuum [title] Plug in the vacuum. [step] Electric vacuums run off of your home's ac power to generate powerful suction. Make sure the vacuum is plugged into a nearby electrical outlet, or that the battery is fully charged.\",\"endings\":\"['Since the dishwasher uses electricity to cook food at room temperature, a blackhead vacuum can probably attach to a large cabinet. [substeps] Turn on your vacuum and wait for the water to drain from the bottom of the vacuum before moving to the next step.', 'If your vacuum has a separate power switch, leave it in the \\\" off \\\" position for now. [substeps] Check for a light display that indicates that your vacuum is connected and charged.', 'If the vacuum is plugged into an outlet, shut the battery off. Press the finish button on the switch inside of the drain for power.', 'If you have one, ensure that it has a soft-head housing with a good connection, and that the cable and tip are aligned perpendicular to the ground. [substeps] Cover any electric outlets with 3 feet (0.91 m) of duct tape.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}