{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"30937201-6ac3-5dae-87ec-a1fb843a68c1","task_key":"default--validation--31460","task_revision_id":"1","upstream_id":"31460","short_description":"HellaSwag validation 31460","config":"default","split":"validation","body":"{\"activity_label\":\"Running a marathon\",\"ctx\":\"A graphic with some keywords forming the number '2' is shown. The scene switches to a black screen with some text about an upcoming marathon. a man\",\"endings\":\"['jumps onto a track with a long javelin and chalks his score, then he dismounts and runs down the track to catch the javelin.', 'is shown jogging through a city on some sidewalks, a bridge, some paths in a park, etc, with a counter on the side showing his progress in kilometers as he goes.', 'sock is seen with peoples name on it and videotape is shown on its surface.', 'celebrates and then is seen on his knees on a balance beam.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}