{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7227a3a2-a827-5d35-a530-992aeffe47ea","task_key":"default--validation--16051","task_revision_id":"1","upstream_id":"16051","short_description":"HellaSwag validation 16051","config":"default","split":"validation","body":"{\"activity_label\":\"Playing harmonica\",\"ctx\":\"A man comes onto the screen thanking his viewers for the comments they have made about his videos. Teams to play the harmonica and perform a song for the video. when he\",\"endings\":\"[\\\"'s done he takes a drink as reporters interfix themselves in between his sailing down the river.\\\", 'is done playing, he stands up and puts his hands in the air and shakes his fist.', \\\"'s done, he comes back on the screen.\\\", 'is done he turns off the camera.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}