{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"47db743e-fb07-56df-9f34-34d2b4cb1d1c","task_key":"default--validation--26190","task_revision_id":"1","upstream_id":"26190","short_description":"HellaSwag validation 26190","config":"default","split":"validation","body":"{\"activity_label\":\"Disc dog\",\"ctx\":\"A man and brown dog are in a fenced off area of an arena with people watching them. The man throws frisbees for the dog to catch. the man\",\"endings\":\"['then kicks an object to the dog holding it, and the dog launches it and catches.', 'retrieves a frisbee from the sandy ground and the dog joins it and dogs it it to catch.', 'performs special tricks with the dog using the frisbee.', 'jumps and twirls the dog across his body very gracefully.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}