{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ec06a434-20ef-5f8c-8159-7bee63320bb2","task_key":"default--validation--4948","task_revision_id":"1","upstream_id":"4948","short_description":"HellaSwag validation 4948","config":"default","split":"validation","body":"{\"activity_label\":\"Spread mulch\",\"ctx\":\"A man is seen speaking to the camera and leads into clips of him moving dirt. other men\",\"endings\":\"['work around him in various angles and ends with their closing poses.', 'help plant the areas while moving around wheel barrows and planting plants while still speaking to the camera.', 'are seen around him as well as an old man picking up rocks and the video ending with the man standing before a group and mowing them down.', 'are seen speaking back and fourth and continue to move around a racetrack.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}