{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"54bffad3-a18d-5efa-9fd8-ede6f73a10bf","task_key":"default--validation--39464","task_revision_id":"1","upstream_id":"39464","short_description":"HellaSwag validation 39464","config":"default","split":"validation","body":"{\"activity_label\":\"Playing harmonica\",\"ctx\":\"He starts to play with his eyes closed and he seems to be really intensely into playing his music. he\",\"endings\":\"['is playing a song, he stops to take a breath for a moment and then continues on with his song.', 'starts to go fast on his harmonica trying to move his hands while he is doing that and then stops for a moment and just starts to play until finally stopping too the own.', 'starts to play faster and harder while he looks down at the camera and keeps playing and rubbing his eyes.', 'makes a fist then takes into his hand several different photos of his songs, songs he has written about his experiences with music and this is driving him crazy.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}