{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"50b669d0-208c-5b9b-ae90-fb95c283480a","task_key":"default--validation--44052","task_revision_id":"1","upstream_id":"44052","short_description":"HellaSwag validation 44052","config":"default","split":"validation","body":"{\"activity_label\":\"Carving jack-o-lanterns\",\"ctx\":\"The inside of the pumpkin gets sprayed with mold remover and patted dry. A candle is placed inside and lit and the lid is put back on. with the lights\",\"endings\":\"['turned on the fire begins to ignite the pumpkin.', 'turned down the jack o lantern can be viewed.', 'on and the pumpkin lit with its protective covering the oil is gently browned and the inside glows until the entire surface glows.', 'turned off, you can still see the wax on the pumpkin and the candle in tongs.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}