{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"953446db-a0d6-572a-9fc5-16401088e302","task_key":"default--validation--17750","task_revision_id":"1","upstream_id":"17750","short_description":"HellaSwag validation 17750","config":"default","split":"validation","body":"{\"activity_label\":\"Running a marathon\",\"ctx\":\"A person is shown in various shots close up as well as leaves and him stretching. the man\",\"endings\":\"['walks away while speaking to the camera and a person holds up a photo.', 'moves several bags and mulch around the tree.', 'shakes his hair to show how it looks and then begins to rub his hands all over it and finally putting some lotion on.', 'is then seen running through a park while looking off into the distance.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}