{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c7299c9f-b40c-5534-8078-c753c8dd469f","task_key":"default--validation--9678","task_revision_id":"1","upstream_id":"9678","short_description":"HellaSwag validation 9678","config":"default","split":"validation","body":"{\"activity_label\":\"High jump\",\"ctx\":\"The man jumps again but this time at 5'10 high, he clears it, and once again it replays. the man\",\"endings\":\"['is now running and jumping over the pole that is now 6 feet high, he clears it like he did all the other heights and once again it replays.', 'performs one last trick on high, this time with an upper hand but he misses again and flies.', \\\"jumps again at 610 and doesn't stick it this time at 611 or 890, he gets a second round and attempts to get off successfully.\\\", 'jumps continuously until it replays to show the height of the spike of the charge and the exact amount of jump result.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}