{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"268c3373-5a5a-5fc6-af68-043f35ff97c0","task_key":"default--validation--34418","task_revision_id":"1","upstream_id":"34418","short_description":"HellaSwag validation 34418","config":"default","split":"validation","body":"{\"activity_label\":\"Installing carpet\",\"ctx\":\"A group of men in black started to removed the first half of the carpet in the living room, then replaced it with lighter, newer carpet. the group of men in black shirts\",\"endings\":\"[\\\"now had their pants down, and they're applying the new carpet to the floor.\\\", 'lay the new carpet onto a yellow cloth over the black cushions and carpet to bring in more color.', 'moved the couch from the wall to other side and removed the other half of the carpet, exposing the green floor.', 'started to extend the carpet and put a new end in between two stands.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}