{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2e833037-8b2d-519f-ac83-2384c57d7c3d","task_key":"default--validation--44049","task_revision_id":"1","upstream_id":"44049","short_description":"HellaSwag validation 44049","config":"default","split":"validation","body":"{\"activity_label\":\"Carving jack-o-lanterns\",\"ctx\":\"First they cut the top off of a green pumpkin and scoop the seeds and other stuff out. The remaining insides are cut and scraped off with an ice cream scoop. a design\",\"endings\":\"[\\\"is added to the pumpkin and they're boxed up and listed for display.\\\", 'is drawn over the inside of a pumpkin and an image and words are drawn along the insides.', 'is then filled in on the pumpkin and the inside with icing and powdered sugar.', 'is drawn on the pumpkin and then cut out.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}