{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"95c539d3-a52b-59b4-82d5-995577fdd388","task_key":"default--validation--39970","task_revision_id":"1","upstream_id":"39970","short_description":"HellaSwag validation 39970","config":"default","split":"validation","body":"{\"activity_label\":\"Surfing\",\"ctx\":\"A close up of shoes are shown that lead into a person putting socks on and waxing off a surf board. the man\",\"endings\":\"['uses a clipper to undo the end and puts the boots back on his feet.', 'is then shown surfing along the water with others as well as landing in the water.', 'is then seen speaking to the camera on a beach while watching various times on the sand.', 'continues waxing the shoes while pausing to give a tutorial on how they should be done.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}