{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8cefa184-665a-5120-987d-ff5a34386e0c","task_key":"default--validation--29818","task_revision_id":"1","upstream_id":"29818","short_description":"HellaSwag validation 29818","config":"default","split":"validation","body":"{\"activity_label\":\"Sharpening knives\",\"ctx\":\"The man then picks up a bottle and squirts some liquid onto one of the objects, and picks up a knife and starts dragging it on both sides along the oil that is on the object. the man\",\"endings\":\"['then puts the dirty object on all four ends of the object.', \\\"then grabs a wrench and alternates using the tool to cut through the paint on the paint until it's gone, then he uses the same object on the square piece of plastic that he closes.\\\", 'then goes to the sink and empties the mixture onto the glass and then turns on the faucet with a cloth.', 'then removes the object and starts rubbing the knife along the hand towel that was under the object and he picks up another tool and begins rubbing the knife along it and once again rubs the knife down with the hand towel.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}