{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3cd1e5dc-3e12-53b3-840f-349b766fbe04","task_key":"default--validation--27969","task_revision_id":"1","upstream_id":"27969","short_description":"HellaSwag validation 27969","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"The man hooks one of the fish on a portable weighing scale. The man drops the fish back into the hole. the man\",\"endings\":\"['pulls the fish out of the hole.', 'weighs the other fish on the portable scale.', 'grabs another fish and slams it in.', 'kneels back down and holds the fish by the tail.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}