{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cf7d094c-737b-5c4f-82b7-b79bdcb3bc97","task_key":"default--validation--27049","task_revision_id":"1","upstream_id":"27049","short_description":"HellaSwag validation 27049","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"A person is seen walking along the ice dragging a sled. The person makes a hole while another walks around and makes holes. the men\",\"endings\":\"['sit around the holes with a pole and one grabs a fish.', 'use the sled to capture fish while the person sits and swims behind him.', 'continue walking and ending with one playing the piano alone in the end.', 'then scale down the cliff and aside from stepping on the ice they continue.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}