{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"dae42fe5-d5bd-5574-b8fb-daea04dd02a7","task_key":"default--validation--46262","task_revision_id":"1","upstream_id":"46262","short_description":"HellaSwag validation 46262","config":"default","split":"validation","body":"{\"activity_label\":\"Doing karate\",\"ctx\":\"A lady and a man in karate uniforms practice kicks in a room with carpet. The lady almost falls trying to kick the man. The cameraman's finger is on the screen. the finger\",\"endings\":\"['stops and then starts again and this man kicks.', 'is get stuck and the man removes it from the camera.', 'is on the camera showing how the ballerina moves her arms to parallel with her body.', 'returns on the screen.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}