{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cd71a3a9-5d67-58ad-b8a4-0aabb6ef94b9","task_key":"default--validation--15043","task_revision_id":"1","upstream_id":"15043","short_description":"HellaSwag validation 15043","config":"default","split":"validation","body":"{\"activity_label\":\"Shaving legs\",\"ctx\":\"A person in a bathroom, with one foot on top of a toilet cover rubs leg lube on their leg. the person\",\"endings\":\"['then begins to shave the length of their leg with a blue razor.', 'pulls the towel from the tap then picks up the solder and iron the other shoe and rubs the shoes to melt the wax.', 'puts the product in their mouth.', 'tips the toe of the shoe upwards and without a sock pulls the leg up.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}