{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5087ee61-bb36-5662-991e-42cf03ee55cd","task_key":"default--validation--19782","task_revision_id":"1","upstream_id":"19782","short_description":"HellaSwag validation 19782","config":"default","split":"validation","body":"{\"activity_label\":\"Having an ice cream\",\"ctx\":\"A lady licks an ice cream cone and holds a baby. The lady takes the pacifier from the baby and lets the baby eat ice cream. the baby\",\"endings\":\"['eats pieces of the ice cream and mixes it in a bowl.', 'makes a face and stare at the camera as she eats.', \\\"isn't happy but he feeds it from the ice cream.\\\", 'is getting more ice cream, and the lady keeps licking his face.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}