{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f803584b-7586-50a2-8969-b57aac53254b","task_key":"default--validation--13877","task_revision_id":"1","upstream_id":"13877","short_description":"HellaSwag validation 13877","config":"default","split":"validation","body":"{\"activity_label\":\"Gargling mouthwash\",\"ctx\":\"A man is shown in a bathroom choosing a toothbrush. He then brushes his teeth and sees blood. he\",\"endings\":\"['undoes the top button of his pants and looks confused.', 'uses listerine mouth wash for thirty seconds to kill the germs.', 'then finishes the brush and straightens his hair.', 'then wets a cloth and scrubs the toothbrush on it.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}