{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2d2cff22-d2cf-5a42-96ee-64aca7f63bd6","task_key":"default--validation--48102","task_revision_id":"1","upstream_id":"48102","short_description":"HellaSwag validation 48102","config":"default","split":"validation","body":"{\"activity_label\":\"Sharpening knives\",\"ctx\":\"A close up of tools are shown as well as a man holding up a knife. The man then begins sharpening the knife using a tool on the table. he\",\"endings\":\"['continues sharpening the knife and showing it close up to the camera.', 'moves the knife around to sharpen further.', 'stops sharpening the knife and leans back in his chair.', 'periodically braids a rope around the knife and begins taking out items attached to the rope.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}