{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9680e1f3-bf23-5b3d-97f8-32d6832ab234","task_key":"default--validation--29776","task_revision_id":"1","upstream_id":"29776","short_description":"HellaSwag validation 29776","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"A man is bending over a hole in the ice, while another is trying to pull out his pole as he has caught something. he\",\"endings\":\"['pulls a bit harder and the other floats up to rescue him before sitting back down.', 'releases the pole and is able to recover it at the bottom of the hole and the man hugs him as they walk away.', 'throws it to the other back and the man catches it as he pushes it in coming back for more.', 'eventually pulls a large trout fish from the hole, holding it up for the camera.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}