{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5a2664fb-8286-5d47-a0bd-72acb192c860","task_key":"default--validation--29415","task_revision_id":"1","upstream_id":"29415","short_description":"HellaSwag validation 29415","config":"default","split":"validation","body":"{\"activity_label\":\"Clean and jerk\",\"ctx\":\"The man drops the weight, pumps his fist, and walks off. The man returns and lifts the weight to his shoulders then over his head. the man\",\"endings\":\"['dismounts and bows his head.', 'resumes lifting until it is released.', 'drops the weight, then lifts it again, and instead of dropping the weight he drops it towards the ground.', 'drops the weight and laughs and pumps his fist.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}