{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"noindex","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[]}},"task_id":"ff95eb2e-ade0-5e69-ac19-a0b7fa32dbc6","task_key":"default--validation--1963","task_revision_id":"2","upstream_id":"1963","short_description":"HellaSwag validation 1963","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"The standing man pulls the pole and wire up higher and the man kneeling on the ice pulls the fish out of the ice water and puts it onto the ice. another man\",\"endings\":\"['in three suits throws the fish down the lake onto the water while the other man at the lower depth touches the fish.', \\\"lays out the fish on the ice while the audience can't make out a face from their turn.\\\", 'is then kneeling on the ice and the men fly up and fly as far as possible.', 'takes a picture of the fish, the kneeling man unhooks it and the standing man picks it up, holds it by its gills and poses with it.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"noindex","subproblems":[],"grids":[]}