{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6e3bc59a-f8c1-5a90-9ff6-78c33f49cb5c","task_key":"default--validation--4744","task_revision_id":"1","upstream_id":"4744","short_description":"HellaSwag validation 4744","config":"default","split":"validation","body":"{\"activity_label\":\"Snatch\",\"ctx\":\"A man in black shirt lifted the barbel and started to carry it up and down for few times, while behind him, two men are removing the metal plates on another barbel. the man in black shirt\",\"endings\":\"['removed the plate and raised the barbel, while people standing next to him held the barbel heads.', 'began to lift the barbel feet and lower them while the people that are standing at three barbel also lifted the barbel, and then lowered it.', 'lifted the barbel and something inside it flipped off and the other man stopped and get the stuff and hand it to the other man who tried to carry it up and handed it to him.', 'added more weight on the barbel by adding yellow plates on both side, then continue weight lifting.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}