{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d3a873bd-07a6-5b97-8623-5927fd30f94c","task_key":"default--validation--7109","task_revision_id":"1","upstream_id":"7109","short_description":"HellaSwag validation 7109","config":"default","split":"validation","body":"{\"activity_label\":\"Elliptical trainer\",\"ctx\":\"Although he is the only one in the gym, there are two people standing behind him at the reception desk. after some time\",\"endings\":\"[', the conversation from earlier comes back to him that it says to leave his cell number with people eventually, so he removes his phone from his ear.', ', the guy begins to workout on the machine and shows its function, while a man walks back and forth into the room from the side door of the gym.', ', he arrives with two young girls and a in a large gym area where many people are standing.', ', the boy appears and starts swinging a basketball on a stick and gets a perplexed look from him.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}