{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e947deab-b8e3-5031-bc48-b02532c15c25","task_key":"default--validation--36592","task_revision_id":"1","upstream_id":"36592","short_description":"HellaSwag validation 36592","config":"default","split":"validation","body":"{\"activity_label\":\"Playing saxophone\",\"ctx\":\"A black male then appears with a green sweater and black hat playing the sax. The man then lays the instrument across his lap and begins talking about what he is doing. after\",\"endings\":\"[\\\"he's finished talking, the look of amazement and disbelief then appears on his face as though he's confused or at least annoyed.\\\", 'he is done talking, he then lays it on the drum.', ', the guy plays some more before finally stopping to talk again.', \\\"he's finished, a band appear and they begin playing putting their hands out when they are done.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}