{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"579e1581-a398-58ff-87a1-76cefc29bc99","task_key":"default--validation--16771","task_revision_id":"1","upstream_id":"16771","short_description":"HellaSwag validation 16771","config":"default","split":"validation","body":"{\"activity_label\":\"Rock climbing\",\"ctx\":\"The man in the green shirt explain the technique as he points to the wall to show the significance of each rock and its height. he\",\"endings\":\"['then grabs a nail clipper and begins showing the technique using his hands.', 'explains how to make sure they shine up.', 'shows various rocks that are named differently and as assigned different numbers.', 'finishes the scene and out we go.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}