{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a6d6fe47-af4d-556a-aee5-c760058adffc","task_key":"default--validation--22914","task_revision_id":"1","upstream_id":"22914","short_description":"HellaSwag validation 22914","config":"default","split":"validation","body":"{\"activity_label\":\"Sharpening knives\",\"ctx\":\"He picks up the sharpening tool and talks about what he will do next and begins to sharpen again. then he\",\"endings\":\"['gets down off the bed and the video ends.', 'gets in the car and drives off of the street.', 'swings the sharpener to sharpen the sharpening stone another time and then takes his knife and scrambles to sharpen another piece.', 'shows all of the pieces that come with the kit and talks about it all.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}