{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7841c9c3-538a-5cbb-853e-095dc0b52c2c","task_key":"default--validation--14078","task_revision_id":"1","upstream_id":"14078","short_description":"HellaSwag validation 14078","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"Finally he uses a long metal hook to hook the fish he caught and brings it out the hole. he\",\"endings\":\"['tightens the hook on the fish to catch it again.', 'pulls a red needle out of the hole and begins going back in to try again trying and pulling some more fish out.', 'then unhooks the long hook and use his tools to unhook the small fishing hook the the fish mouth leaving the fish bloody.', 'fish out of the hole and attempts to catch some more.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}