{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9abd2a73-ba7b-5ce5-b8ae-c9a08ac526aa","task_key":"default--validation--36149","task_revision_id":"1","upstream_id":"36149","short_description":"HellaSwag validation 36149","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to clean an automotive paint gun [title] Remove the air regulator from the paint. [step] The air regulator is the device at the bottom of the gun where compressed air can be hooked up to the bottom of the air gun. Make sure that the compressed air source is disconnected from the regulator first and then you can disconnect the regulator from the gun.\",\"endings\":\"['[title] Open the lid from the paint cup to pour out any unused material. [step] You need to make sure you dispose of the waste in the proper containers.', '[substeps] If you do not have a regulator, you can purchase one. These usually come with a valve on the bottom of the gun that you tighten.', '[substeps] For fuel gas fuel fuel [title] Remove the gas filter. [step] To get gas gas fuel can, remove the gas filter located on the underside of the oil or oil oil safe oil pan.', 'You can loosen the air regulator manually if it is stuck. [title] Clean the interior of the gun with a damp cloth.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}