{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"11f74e20-575d-5c1a-a072-4bdcf88a927c","task_key":"default--validation--49519","task_revision_id":"1","upstream_id":"49519","short_description":"HellaSwag validation 49519","config":"default","split":"validation","body":"{\"activity_label\":\"Playing harmonica\",\"ctx\":\"The man then begins playing again using his hands to indicate the type of note he is hitting. the man\",\"endings\":\"['sticks his hands directly down on the second note.', 'continues playing with his phone and a few more shots of him and spectators are shown as well as people winning and still playing.', 'then begins playing again before stopping and talking to the camera one last time.', 'continues to hit the object over and over again while the camera pans around the area.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}