{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"87ea0fc1-31a4-5ff3-83e4-7b2f8b186a70","task_key":"default--validation--2766","task_revision_id":"1","upstream_id":"2766","short_description":"HellaSwag validation 2766","config":"default","split":"validation","body":"{\"activity_label\":\"Hanging wallpaper\",\"ctx\":\"A break is needed and the woman get a drink from the kitchen. Shortly after, the lady continues her tasks and then walks outside to take her cat out. the wall\",\"endings\":\"[\\\"still isn't finished and she goes back to the room to finish it rolling the outside of it to ensure its smoothness.\\\", 'goes up and the roach and garden are shown.', 'then fades to black and the camera are seen from several angles.', 'slides down, showing to the camera the seeping hole that the hole in the floor has made.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}