{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8c46afee-e776-5714-9317-cce0fbc982c3","task_key":"default--validation--18206","task_revision_id":"1","upstream_id":"18206","short_description":"HellaSwag validation 18206","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to use a gas can [title] Place the can on the ground next to a gas pump station. [step] Gasoline often carries a static electric charge. Place the can on the ground before filling it, where any static electric charge carried by the gasoline can disperse.\",\"endings\":\"[\\\"[substeps] To check if the gas can is on the ground, either walk or stand into the gas's fill station. Test the gas can for static shock by dropping the lid on and shaking it.\\\", '[title] Take the can opener from the oil pan, give it a quick shake. [step] The gun should pulse warm to the touch, without the blade rattling.', 'If you fill the can on a plastic surface, sparks may occur and ignite the gasoline. [title] Place the gas pump nozzle into the top of the can.', 'Wear gloves and eye protection when handling an electric can. [title] Use the gas can just inside the gas station.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}