{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a19af980-5842-52e1-80e4-4a383ba09905","task_key":"default--validation--41180","task_revision_id":"1","upstream_id":"41180","short_description":"HellaSwag validation 41180","config":"default","split":"validation","body":"{\"activity_label\":\"Running a marathon\",\"ctx\":\"They both try to get up but they keep falling down. One of the women starts to crawl to the finish line. The other woman follows her. a man\",\"endings\":\"['lifts one of the women and hugs her.', 'comes on and starts using his shovel to help start the fire.', 'rides his bicycle after the last runners get down.', 'squats on a pit of dirt.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}