{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5391c45d-1518-5c70-98ef-a06641e639a1","task_key":"default--validation--36044","task_revision_id":"1","upstream_id":"36044","short_description":"HellaSwag validation 36044","config":"default","split":"validation","body":"{\"activity_label\":\"Sharpening knives\",\"ctx\":\"A still shot is shown as the word honing is on the screen and music plays in the background. he\",\"endings\":\"['is now entering a building and is talking about vacuuming the floor.', 'has equipment and tools for shoots, some above ground, others smoke and still shots of himself taking off with his shot put.', 'finally uses a hammer to hammer the window and it sticks in place with ease.', 'explains how to hold a honing steel.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}