{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8a4557e5-0341-5776-865a-7c5fe09095f7","task_key":"default--validation--22687","task_revision_id":"1","upstream_id":"22687","short_description":"HellaSwag validation 22687","config":"default","split":"validation","body":"{\"activity_label\":\"Sharpening knives\",\"ctx\":\"A man is giving advice on outdoor activities, specifically mushrooming and closeup is shown and the man talks about it. The man cuts one open and then cuts it in half. another man\",\"endings\":\"['is shown while he is standing on a ladder trimming the fence while he talks.', 'talks about the experience.', 'follows suit to cut the wooden pieces and then cuts them all and continues talking and afterwards squeezing some out.', 'is shown doing the same thing and the first man begins to back pedal and cuts one again.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}