{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5abf0738-69b9-59f2-afd0-0c8ed4883db8","task_key":"default--validation--21742","task_revision_id":"1","upstream_id":"21742","short_description":"HellaSwag validation 21742","config":"default","split":"validation","body":"{\"activity_label\":\"Scuba diving\",\"ctx\":\"Several scuba divers are seen swimming around the ocean while sharks move around them underneath. the camera\",\"endings\":\"['follows the scuba divers as they look on at their work.', 'pans out and reveals many boats and people floating around.', 'continuously follows the sharks around the water and the scuba divers interact with one another while they move.', 'pans around the divers jumping as they swim around the area.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}