{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"noindex","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[]}},"task_id":"569fb807-3865-5530-aeb5-63fcae7d6537","task_key":"default--validation--2581","task_revision_id":"2","upstream_id":"2581","short_description":"HellaSwag validation 2581","config":"default","split":"validation","body":"{\"activity_label\":\"Hand washing clothes\",\"ctx\":\"A young man is washing clothes inside a bathroom sink as he talks to the camera. initially he\",\"endings\":\"['starts to wash the clothes with both hands.', 'shows several different ways to clean a sink during a wash and is seen staring into the camera.', 'is demonstrating using two hoses to wash the clothes as he will talk to the camera.', 'is going to his locker but resumes washing the shirt.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"noindex","subproblems":[],"grids":[]}