{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a016abb0-036c-5a7a-b24d-20580c7f3a0c","task_key":"default--validation--11265","task_revision_id":"1","upstream_id":"11265","short_description":"HellaSwag validation 11265","config":"default","split":"validation","body":"{\"activity_label\":\"Skiing\",\"ctx\":\"A mountain full of snow appears and a man begins skiing down the slopes. as he\",\"endings\":\"['skiis, a small mountain gorilla appears.', 'puts distance between them, another man joins him and begins skiing and falling off.', 'skis, more men appear before he reappears by himself.', 'slides down he lifts his arm up and brings it down to reveal to be broken and covered in blood.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}