{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b605a521-53c9-5868-9bf7-d716478a1e06","task_key":"default--validation--24493","task_revision_id":"1","upstream_id":"24493","short_description":"HellaSwag validation 24493","config":"default","split":"validation","body":"{\"activity_label\":\"Applying sunscreen\",\"ctx\":\"There is a boy sitting on the bed showing off his sunburn and putting cream on it. A friend is shaking his head because of the other guy's sunburn. the friend\",\"endings\":\"['holds a little pin he is holding up and pushes it in the sunburn.', 'puts his shirt on, gets up and walks out of the room again and another boy is showing the same sunburn.', 'puts a bottle of suntan lotion and rubs it on the legs of the man.', 'lifts his shirt to show his sunburn as well.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}