{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6d17d919-ad17-54fa-add8-1ba5c43f7552","task_key":"default--validation--38185","task_revision_id":"1","upstream_id":"38185","short_description":"HellaSwag validation 38185","config":"default","split":"validation","body":"{\"activity_label\":\"Having an ice cream\",\"ctx\":\"A coffee mug sits on a counter then dry ingredients and water are added before the contents are stirred with a spoon. A graduated cup with milk has the mixture poured into it and is stirred. a bowl\",\"endings\":\"['has cream and a chocolate mixture added in and blended together.', 'with vodka is added next and onions are added then the blender which is used before it is clasped and stirred vigorously.', 'is filled with ice and tossed out behind the instance.', 'of sugar are also added while not stirred.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}