{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"04347a51-1752-5204-96a2-8b83c0451d4f","task_key":"default--validation--28165","task_revision_id":"1","upstream_id":"28165","short_description":"HellaSwag validation 28165","config":"default","split":"validation","body":"{\"activity_label\":\"Disc dog\",\"ctx\":\"A person throws a frisbee and the dog runs to get it. The man lays down and put his feet up, and the dog jumps on the feet. then\",\"endings\":\"[', the man throws the frisbee and the dog runs to get it.', ', people run to bag the frisbee, and the people are standing behind the man talking.', ', the man puts his feet down and a dog jumped on his feet.', ', the man the man toss the frisbee and the dog runs to fetch it.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}