{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"18bfa9bd-8608-5d56-8394-6605ba12d626","task_key":"default--validation--47704","task_revision_id":"1","upstream_id":"47704","short_description":"HellaSwag validation 47704","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"A camera pans around several snow mobiles and shows people sitting around a hole. A man swims up in the water and speaks to the camera. another man\",\"endings\":\"['is water skiing and showing several different tricks and how to do various flips into the depth.', 'comes out of the water wet.', 'climbs into the water next to him and they begin skiing.', 'sits while he speaks and holds a frisbee already hanging from his saddle.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}