{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d2397858-384e-5dc6-b90e-af9d22ea9d38","task_key":"default--validation--31374","task_revision_id":"1","upstream_id":"31374","short_description":"HellaSwag validation 31374","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"The man grabs his fishing pole and starts fishing in the hole in the ice. The man appears to have caught something and kneels down to pull his fishing line by hand. the man\",\"endings\":\"['releases the fishing line and drops the fishing pole.', 'grabs his pole and pulls several fish out of the hole.', 'stops fishing and emerges to speak to the camera.', 'pulls a fish out of the water and poses with it.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}