{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4df10707-061f-502c-ac50-6b47ffc2e6c3","task_key":"default--validation--39408","task_revision_id":"1","upstream_id":"39408","short_description":"HellaSwag validation 39408","config":"default","split":"validation","body":"{\"activity_label\":\"Calf roping\",\"ctx\":\"A man is seen riding a horse onto a field while swinging a rope and a young calf running in front of him. the man\",\"endings\":\"['then throws the rope onto the calf and ties to animal up then walking away.', 'rides the horse around while the calf catches up and runs away.', 'throws the rope towards the calf and the calf runs after it.', 'competes several tricks on the horse and ends by holding the calf down and marching him off the field.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}