{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7795dfc4-7646-53e0-aea3-0772fce92cdf","task_key":"default--validation--21409","task_revision_id":"1","upstream_id":"21409","short_description":"HellaSwag validation 21409","config":"default","split":"validation","body":"{\"activity_label\":\"Gargling mouthwash\",\"ctx\":\"They are swishing it in their mouths and almost all of them have their hands over their mouths. One of them spits it out in the sink and the other three are trying to carry on. they\",\"endings\":\"['are now following another man driving a car that sprays cleaner all over the car and he is shaking and smeared with the solution as well as laughing and bordering on spitting out some of it too.', 'try to talk to each other and take their fingers out of their mouths to talk.', 'all seem to be pretty groomers.', 'are having a really hard time and eventually they all spit out and one girl wins.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}