{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8ef4b109-715e-57b8-81fa-83fc23e8e85c","task_key":"default--validation--23823","task_revision_id":"1","upstream_id":"23823","short_description":"HellaSwag validation 23823","config":"default","split":"validation","body":"{\"activity_label\":\"Clean and jerk\",\"ctx\":\"Two men in work out clothes are standing up against the wall and they begin to talk about working out. one man\",\"endings\":\"['then goes around the room and bends down and uses the barbell to lift himself up off the floor while the other men look at him and laugh.', \\\"then comes over and takes his turn and he's wearing a helmet.\\\", 'is then shown and he bends down and shows the proper way to lift the bar with weight on it.', 'wraps his hand around his injured arm and the other man holds his arm while he watches him start.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}