{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6bb8a505-fa12-541d-87a5-d9264bc8849d","task_key":"default--validation--23899","task_revision_id":"1","upstream_id":"23899","short_description":"HellaSwag validation 23899","config":"default","split":"validation","body":"{\"activity_label\":\"Slacklining\",\"ctx\":\"A tight rope is strung across the length of the pool and a woman is attempting to walk across, she falls several times. The last time she falls, the camera rewinds and shows her falling backwards. the video\",\"endings\":\"['then cuts to the woman exercising in four different positions, and showing off her various tricks.', 'ends with the closing captions shown on the screen.', 'ends with the closing captions on the screen.', 'then cuts to a man who is also using a tight rope to cross a pool.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}