{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6f282ccf-d431-5b02-97b9-877de6c62219","task_key":"default--validation--18166","task_revision_id":"1","upstream_id":"18166","short_description":"HellaSwag validation 18166","config":"default","split":"validation","body":"{\"activity_label\":\"Cleaning windows\",\"ctx\":\"As he does this he explains his methods. Using the same rag he moves to the other pane to clean the frame. finally\",\"endings\":\"[', he turns around and begins rubbing the umbrella on the window.', ', the water sports appears and he sprays it with water before shooting his glass and setting it in the trashcan.', 'he cuts the tape on he window and gets a custom paste and paints the window with it.', ', he ends by scrubbing the bottom right corner of the door.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}