{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"32449508-f2a4-5647-8a88-f9c9ce4f30ec","task_key":"default--validation--27796","task_revision_id":"1","upstream_id":"27796","short_description":"HellaSwag validation 27796","config":"default","split":"validation","body":"{\"activity_label\":\"Sharpening knives\",\"ctx\":\"A woman shows a red edge grip while talking. Then, she takes a knife and put it on the cleft of the edge grip and continues talking. after\",\"endings\":\"[', the woman pass the knife on the edge grip from front to back while explaining.', ', she writes down instructions the point turn of the knife to adjust the pommel.', ', the woman puts out the nut around.', ', the woman uses a measuring tape to trim the crotch of the lady.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}