{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"947fe7c5-55d0-554d-9d80-f616fdcde297","task_key":"default--validation--18727","task_revision_id":"1","upstream_id":"18727","short_description":"HellaSwag validation 18727","config":"default","split":"validation","body":"{\"activity_label\":\"Sharpening knives\",\"ctx\":\"The man adds another block on the screen and puts water on the block. The man begins to sharpen the knife again. the man\",\"endings\":\"['finishes finished sharpening the knife.', 'points to the board then up to the board and continues to sharpen the knife.', 'stops and shows the knife.', 'then puts salt on the board and adds the different block in a circle.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}